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1.0  INTRODUCTION 

1.1  Purpose  of  the  Report 


This  report  has  been  produced  In  response  to  two  paragraphs 
In  the*  Statement  of  Work  (for  Contract  iDCA100-84-c-0085, 
"Analysis  and  Resolution  of  Packet  Switching  Issues")  which 
read  as  folloiis: 

2.3  Next  Generation  Packet  Switch 

As  higher  bandwidth  long-haul  (satellites,  optical 
fiber)  and  short-haul  (broad-band  cable,  microwave) 
transmission  media  become  cost-effective  for  DDN 
trunking,  and  as  high  bandwidth  host  applications 
becoBM  more  widespread,  a  radically  different  packet 
switch  will  be  needed  for  the  ODH  functions  performed 
by  hardware  or  firmware.  A  genuine  need  for  a  new 
switch  must  be  forecast  far  enough  In  advance  to  permit 
development  and  testing. 

4.3  Identify  the  Requltesttnts  for  the  Next  Generation 
DON  Packet  switch 

The  contractor  shall  perform  the  research  and  necessary 
analyses,  and  prepare  a  report  recommending  require¬ 
ments  for  the  next  generation  DDN  packet  switch, 
including  functional,  reliability,  survivability, 
throughput,  maintainability,  and  security  requirements. 
The  report  shall  predict  dates  by  which  the  C/30  will 
begin  to  fall  seriously  short  of  the  DDN  requirements, 
estimate  when  the  new  packet  switch  should  be  made 
available,  and  estimate  the  length  of  time  required  for 
system  development  and  testing.  The  requirements 
developed  shall  be  based  on:  the  potential  use  of 
alternative  trunking  facilities  in  the  DDN,  such  as 
satellite  circuits  and  local  wideband  distribution 
systems;  the  increasing  use  of  high  bandwidth  host-to- 
host  applications;  and  the  role  of  the  DDN  In  an 
Internetworking  system. 


The  Intended  audience  and  users  of  this  report  Is  the  De¬ 
fense  Data  Network  (DDN)  and  those  Involved  In  the  develop¬ 
ment  of  packet  switched  networks  for  the  Department  of 
Defense  (DoD).  The  reader  Is  assumed  to  have  some  knowledge 
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o£  networking,  o£  packet  technology,  o£  the  DoD  Internetwork 
structure  and  familiarity  with  computer/communications 
hardware  and  with  real-time  software  for  communications. 
This  report  is  expected  to  be  used  by: 

the  Government  to  Include  with  a  procurement 
specification  for  the  design  of  the  next  generation 
packet  switch 

the  Government  to  include  with  a  procurement  speci¬ 
fication  for  the  development  of  the  next  generation 
packet  switch 

the  contractors  who  bid  on  these  procurements  in  order 
to  correctly  size  the  scope  of  the  work 

the  contractors  who  perform  on  these  procurements  as 
guidance  for  their  more  detailed  design  and  development 

As  noted  above,  this  report  assumes  a  general  understanding 
of  the  concepts  of  packet  switching.  This  understanding  can 
be  obtained  from  a  collection  of  papers  such  as  the  DARPA 
compendium  (DAR  81]  or  the  November  1978  issue  of  the  IEEE 
Proceedings  CIBI  78].  The  fundamental  concepts  of  packet 
switching  data  communications  can  be  summarized  as: 

messages  are  broken  up  into  packets  (originally 
typically  about  1000  bits)  which  individually  have  a 
high  probability  of  traversing  a  link  without  errors 

the  nodes  of  the  network  are  computers;  they  can 
perform  "intelligent”  functions  and  they  can  provide 
storage  for  messages/packets  until  their  receipt  is 
acknowledged 

overall  reliability  is  achieved  by  making  use  of  error 
detection  and  requesting  retransmission 

As  the  ODN  evolves  and  grows  during  a  time  of  rapidly  chang¬ 
ing  needs  and  technology,  careful  planning  and  development 
will  produce  the  Next  Generation  Packet  Switch  (NOPS). 
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1.2  Alternative  DDN  Environments 

In  preparing  this  report  attention  was  paid  to  the  evolving 
technologies  and  to  comaMrcial  developments  in  data  and 
voice  communication.  The  purpose  of  this  was  to  determine 
how,  and  to  what  extent,  these  considerations  %K>uld  affect 
the  Next  generation  Packet  Switch  (MOPS)  and/or  its  environ¬ 
ment.  The  conclusion,  discussed  in  Section  5,  was  that  the 
MOPS  environment  would  be  an  upward  evolution  of  the  present 
environment  and  not  a  radical  departure.  Alternative  DDH 
environments  were  considered  briefly,  however;  these  are 
described  in  Appendix  B-  since  we  feel  that  soma  of  the 
alternatives  may  have  a  role  to  play  in  a  following 
generation  of  the  evolving  DM. 
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1.3  Report  organization 

This  report  la  organized  into  eleven  sections  and  two  appen¬ 
dices;  the  Introduction  Is  section  1.  Section  2  susMarlzes 
the  current,  near-tera  future  status,  and  llsiits  to  growth 
of  the  DDM.  Section  3  discusses  the  expected  growth  and 
changing  requlreaents  for  the  DOM.  Section  4  examines  tech¬ 
nologies  which  impinge  on  the  future  DOM.  Section  5  postu¬ 
lates  the  environment  for  the  MOPS.  Section  6  is  a  discus¬ 
sion  of  the  requirements  for  the  MOPS.  Section  7  revie%» 
some  commercial  packet  switch  developments  and  assesses 
their  potential  for  meeting  the  MOPS  requirements.  Section 
8  presents  some  strategies  for  acquiring  the  MOPS.  A 
suggested  schedule  for  the  M<V8  is  given  in  Section  9.  Fi¬ 
nally  a  set  of  recoamMndations  for  the  MOPS  is  provided  in 
Section  10.  References  for  the  report  are  in  Section  11. 
Appendix  A  provides  a  more  detailed  discussion  of  how  the 
requirements  for  the  MOPS  were  developed,  including  some 
discussion  of  multiprocessor  architectures  as  they  fit  into 
the  HOPS  application.  Appendix  B,  as  noted  in  Section  1.2 
above,  provides  a  brief  discussion  of  some  alternative  net¬ 
work  environments  to  the  environment  chosen  in  Section  5. 
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2.0  WHERE  DON  IS  AMD  NEAR  TERM  FUTURE 


2.1  ARPAMET-DDN  Evolution 

We  shall  examine  the  current  DDN  and  Its  past  and  future 
evolution  as  «re  forecast  the  major  changes  that  will  arise 
over  the  next  decade.  These  changes  are  of  two  types:  those 
due  to  specific  DoD  gro%rth  In  needs  and  usage;  and  those 
driven  by  growth  In  data  communication  which  are  largely  a 
result  of  the  changes  In  computing  and  communication 
technology.  The  current  and  near  term  projected  DDN  Is  a 
direct  evolution  of  the  ARPANET  (described  In  (DAR  611).  It 
Incorporates  part  of  the  original  ARPANET  as  MILNET  and 
additions  built  out  of  the  ARPANET  technology.  DDN  plans 
through  FY  66  are  described  in  the  DDN  Program  Plan  (DDN  621 
and  In  a  paper  by  Helden  and  Duffleld  (HEX  621.  Major 
changes  face  DDE  as  It  moves  away  from  the  original  class  of 
research  applications  Into  changing  network  technology  and 
rapidly  growing  operational  applications  for  the  Department 
of  Defense. 

2.2  The  ARPANET  Technology 


A  brief  discussion  of  the  ■  historical  evolution  of  the 
ARPANET  Is  useful  In  understanding  how  the  current  DDN  was 
arrived  at.  The  ARPANET  was  Initially  a  research  project 
Intended  to  develop  the  advantages  of  a  distributed  packet 
switched  network  for  reliability  and  efficiency  of 
operation.  Baran  (BAR  64]  proposed  that  reliability  and 
survivability  could  be  achieved  by  having  multiple  source- 
destination  routes  through  a  distributed,  highly-connected 
network  of  nodes  (switches)  and  trunks.  Data  communication 
efficiency  was  to  be  obtained  by  using  packets  to  provide  a 
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specialized  time-division  multiplexing  for  "bursty"  traffic 
on  high  speed  data  links. 

2.2.1  The  Packet  Switches 

The  first  generation  packet  switches  were  produced  by  Bolt 
Beranek  and  Netrman  (BBM)  as  specially  programmed  versions  of 
the  Honey%rell  516  minicomputer;  they  are  described  by  Heart 
(HBA  701  and  were  called  Interface  Message  Processors 
(IMPS).  The  network  was  provided  by  connecting  hosts  to 
IMPS  with  access  lines  and  IMPs  to  IMPs  over  trunks  making 
up  the  distributed  network.  Individual  terminals  were 
originally  connected  through  their  local  hosts.  Later 
terminals  were  connected  directly  through  Terminal  Access 
Controllers  (TACs);  which  used  the  Honeywell  316 
minicomputer  and  served  as  a  combination  of  a  local  host  and 
an  IMP  (ORN  721.  The  IMPs  %rere  programmed  to  carry  out  the 
functions  of  communication  with  hosts  and  each  other  using  a 
set  of  electrical  interfaces  and  communications  protocols 
which  have  continued  to  evolve  over  time.  Both  the  IMP  and 
the  TAC  were  uniprocessors. 

In  the  late  1970s  a  new  multiprocessor  packet  switch,  the 
PLURIBUS-IMP,  commonly  called  PLURIBUS  [KAT  781  was 
designed  and  constructed  by  BBN  in  relatively  limited 
numbers;  the  individual  processors  were  Lockheed  SUB 
minicomputers.  As  a  multiprocessor,  the  PLURIBUS  had  higher 
overall  throughput;  it  was  also  designed  to  have  fault- 
tolerance  and  to  be  capable  of  fail-soft  operation. 
PLURIBUS  switches  have  been  used  on  specific,  primarily 
satellite,  high  data  rate  links  [LIN  79].  Study  of  the 
special  requirements  of  satellite  links  led  to  a  proposed 
upgrade  of  the  PLURIBUS  using  a  reengineered  and  faster 
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version  of  the  SUB  (NBL  61].  This  packet  switch  was  never 
Implemented . 

More  recently  BBM  has  built  an  IHP-upward-con^atlble  packet 
switch  called  the  C/30  (HAV  82].  This  uniprocessor  Is 
microcoded  to  Implement  the  basic  Honeywell  xl6  instructions 
and  some  common  sequences  of  those  Instructions.  The  C/30 
also  accommodates  a  larger  address  space,  isote  communication 
ports,  and  utilizes  a  much  higher  level  of  integration  In 
both  logic  and  memories.  For  high  reliability  of  network 
service  a  host  may  be  connected  to  two  different  switches; 
this  connection  is  referred  to  as  "dual  homing". 

2.2.2  The  Communication  Links 

The  original  ARPANET  Internode  trunks  are  nominal  50  kilobit 
per  second  (kbps)  terrestrial  lines  leased  from  commercial 
communication  carriers  as  are  many  of  the  current  DON  com¬ 
munication  trunks  Other  lines  are  generally  at  9.6  kbps. 
These  lines  have  an  error  rate  of  about  1  in  10,000,000, 
thus  most  packets  of  lengths  on  the  order  of  1000  bits  will 
transit  a  concatenation  of  links  and  be  error  free.  The 
time  for  a  packet  to  completely  traverse  such  a  terrestrial 
link  is  almost  totally  made  up  of  the  time  required  by  the 
duration  of  the  packet  at  the  link  speed.  For  example,  on  a 
link  1000  miles  long  the  time  required  for  an  individual  bit 
to  traverse  the  link  is  about  1  msec  while  the  transit  time 
of  a  1000  bit  packet  at  50  kbps  is  20  msec.  Thus  transit 
times  dominate  propagation  delays  for  terrestrial  links  of 
1000  or  2000  miles. 
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2.2.3  The  PSN  Protocols 

ARPANET  packet  switches  provided  logical  host  access 
protocol  on  three  different  electrical  interfaces.  These 
were  the  Local  Host  (LH),  Distant  Host  (DH),  and  Very 
Distant  HOST  (VDH)  interfaces.  A  fourth  interface,  HDLC 
Distant  Host  (HDH)  %fas  recently  added  %idiich  allows  use  of  a 
standard  line  protocol.  ARPANET *s  original  Host  Access 
protocol  was  kno%Ri  as  1822;  it  is  now  knom  as  the  ARPANET 
Host  Interface  Protocol  (AHIP).  The  DDN  now  also  supports 
an  alternate  host  protocol,  X.25.  The  links  betmen  PSNs 
originally  used  the  Binary  Synchronous  Coamunication 
protocol  (BSC).  This  has  been  upgraded  to  a  bit-synchronous 
protocol,  HDLC. 

2.2.4  Security 

ARPANET  originally  had  no  provision  for  security.  At  a 
later  tine  experinental  and  United  operational  coanunica- 
tion  security  was  applied  to  selected  data  transnission  over 
the  ARPANET  using  the  Private  Line  Interface  (PLI).  The  PLI 
consisted  of  two  niniconputers  and  a  cryptographic  device. 
The  two  niniconputers  were  used  to  provide  separate 
processing  for  the  "Red**  or  clear  text  infornation  and  the 
"Black"  ox  encrypted  Infornation.  A  nininun  of  destination 
indicative  infornation  is  passed  between  the  two 
niniconputers  and  the  renainder  of  the  infornation  nust  pass 
through  the  cryptographic  device  (VAL  821. 

2.2.5  Sunnary 

ARPANET  used  a  conbination  of  progrannable  conputers  with 
connunication  posts  and  high  quality  leased  lines  to  build  a 
packet  switching  net%rosk.  The  suite  of  connunication 
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protocols  and  the  functionality  of  the  packet  switches 
evolved  with  experience  and  need.  As  the  network  grew,  both 
In  size  and  traffic  deaand,  switches  %Mre  upgraded  and 
coMBunicatlon  links  %iere  added.  The  basic  switch  archi¬ 
tecture  was  enhanced  and  the  softtmre  has  gro%m  in  an  up%Mird 
conpatible  way.  In  the  next  section  examine  DDH  and  its 
near  term  gro«fth  potential. 

2.3  DON  Mow  and  Hear  Term 

The  1986  DDM  is  described  in  the  (Draft)  White  Paper  on  DDM 
Capacity  (PRI  851  from  which  soma  of  the  following  material 
is  dram.  At  this  time,  the  plans  are  for  two  separate 
networks,  MXLMtT,  %#hich  is  unclassified  and  DISHIT,  which 
will  be  classified.  The  1986  MXLMIT  will  have  174  Packet 
Switching  Modes  (PSMs)  and  300  tranks. 

2.3.1  The  Packet  Switches 

The  C/30  PSMs  are  being  upgraded  to  the  C/301  which  can 
logically  support  44  connections.  A  host  represents  one 
connection  and  a  trunk  represents  two  connections;  software 
limits  the  number  of  trunks  to  14.  There  are  some  current 
logical  limitations  which  can  be  overcome.  These  deal  with 
addressing  of  nodes  and  the  number  of  ports  per  node. 
Originally  the  number  of  packet  switch  nodes  (PSMs)  was 
restricted  to  253.  The  new  IHP  Ind-to-end  Protocol  (see 

2.2.3  below)  will  raise  that  to  1024.  In  order  to 
accommodate  individual  terminals  TACs  have  been  used;  each 
can  accommodate  63  terminals.  A  new  device,  the  Met%rork 
Access  Controller  (MAC)  (ILD  831,  will  go  into  service  this 
year;  one  of  the  operational  modes  of  the  MAC  is  as  a  "mini- 
TAC"  trhich  can  act  as  a  concentrator  for  16  terminals. 
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2.3.2  Trunks  and  Access  Lines 

There  have  been  no  Mjor  changes  in  the  trunks  and  access 
lines  as  ARPAMIT  has  evolved  Into  DDH.  The  Mjority  of  the 
trunks  are  high  speed  (50  or  56  kbps);  the  reasinder  are  at 
9.6  kbps.  Access  lines  provide  speeds  from  1.2  kbps  to  56 
kbps. 


2.3.3  The  P8H  Protocols 

The  cosBMinication  protocols  within  the  DDM  have  had  a 
continuing  evolution  since  the  DDN  was  initiated.  The 
latest  version  (Release  7.0)  will  contain  a  aajor  change  to 
the  POM  softuare,  the  new  Ind-to-Bnd  (BB)  protocol  for  the 
current  packet  switches  (MAL  OOa,  MAL  84b,  HAL  861.  This 
protocol  establishes  duplex  connections  between  endpoint 
POMs  (the  POMs  connected  to  the  hosts  which  are  coaanini- 
cating).  This  change  will  iaprove  POM  and  network 
throughput  by  reducing  the  nuidNir  of  purely  "administrative** 
Messages  that  support  reliable  operation.  Other  enhance¬ 
ments  in  Release  7.0  include  support  for  satellite  links  by 
providing  adjustable  windows,  support  for  precedence  and 
preemption,  and  interoperable  AHIP-DDN  X.25  service. 

In  release  8.0  the  BB  protocol  will  provide  even  more 
service;  they  will  include  fragmentation  and  reassembly  of 
datagrams.  This  will  be  done  with  less  overhead,  but 
without  total  reliability  which,  if  needed,  would  have  to  be 
provided  by  a  higher  level  protocol. 


2.3.4  Security 


The  current  security  architecture  for  the  DOM  is  docuamnted 
for  the  subscriber  in  the  DDM  Subscriber  Security  Guide 
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(8HX  831  and  its  current  new  draft  ISHX  861.  It  Is  also  de¬ 
scribed  in  the  new  draft  of  DDN's  future  security  archi¬ 
tecture  (ODM  851.  Briefly,  the  ODN  is  divided  into  two 
se^swnts,  one  classified  and  the  other  unclassified.  Both 
se^aents  use  the  C/3x  PSMs  and  trunk  security  will  be 
provided  by  link  encryption  with  KO-84AS  for  both  segaents; 
the  link  keying  aster ial  on  DISMBT  will  be  protected  as 
SBCRBT.  Switches  at  DIBNBT  sites  will  be  physically 
protected  at  the  8BC31BT  level.  DISMBT  access  lines  will 
also  be  protected  by  KO-84AS  and  MXLMBT  access  lines  will  be 
protected  by  Low-Cost-Bncryption  Authentication  Devices 
(LBADsl.  Subscribers  will  be  able  to  have  additional  end- 
to-end  encryption  (13)  protection  by  using  BLACKER  Front 
Bnds  (BPBsl  which  can  provide  flexible  and  dynaaic  B3 
protection. 

2.4  Prospective  Changes  to  the  DDM 

There  are  a  nuaber  of  prospective  changes  to  the  DDM  which 
bear  on  the  capacity  and  throughput  of  the  network  and  the 
PSMs.  One  such  change  deals  with  iapleasnting  a  congestion 
control  aechanisa  (BIS  851.  Such  a  aechanisa  will  iaprove 
throughput  when  the  network  is  heavily  loaded.  In  the  past, 
especially  on  ARPAMBT  which  was  always  lightly  loaded,  there 
were  no  aajor  congestion  probleas.  With  congestion  control 
procedures  in  place,  one  can  feel  aore  certain  about 
throughput  calculations. 

Another  prospective  change  deals  with  Type-of-Service  (TOS) 
routing  (OAR  851.  TOS  routing  takes  advantage  of  the  fact 
that  it  is  appropriate  to  route  different  sorts  of  transais- 
sions  over  different  paths.  For  exaaple,  file  transfers  are 
suitable  for  transaission  over  satellite  trunks  with  high 
bandwidth  and  (relatively)  long  delay.  Single  packet 
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transBisslons,  on  the  other  hand,  benefit  froa  low  delay 
paths.  T08  routing,  of  course.  Interacts  with  other 
protocol  features  and  changes. 

Still  another  prospective  change  which  can  affect  DDM 
capacity  is  the  expected  availability  of  the  BBM  c/300 
packet  switch  at  the  end  of  1986.  This  P8M  will  have  twice 
the  MMory  of  the  C/30B,  the  ability  to  support  64  (vice  44) 
attached  devices,  and  increased  throughput.  It  can  operate 
in  a  network  with  C/30s  and  C/30Bs.  The  White  Paper  on  DDM 
Capacity  (PRl  851  aakes  a  nuB8>er  of  its  calculations  under 
the  assuaption  that  the  C/300  will  be  placed  into  service 
for  DDM.  Since  the  situation  regarding  the  C/300  is  un¬ 
clear,  %ie  shall  develop  the  date  by  «fhich  the  MOPS  is 
requited  under  both  conditions:  with  and  without  the  C/300. 

2.5  SuMMry  and  Conclusion 

The  DDM  and  its  PSNs  represent  an  upward  evolution  of  the 
ARPAMIT  technology.  Architecturally,  the  PSMs  (except  for 
the  PLURIBU8  and  successor  satellite  nodes)  are  still  uni¬ 
processors.  Reliability  of  PSMs  is  achieved  by  the  inherent 
reliability  of  the  cosq>uters  thesaelves,  by  the  redundancy 
provided  in  the  topology  of  the  DDM,  and  by  dual  homing  of 
critical  hosts.  The  network  protocols  have  undergone  and 
will  continue  to  undergo  change.  DDM  is  growing  at  about 
20%  per  year.  Sosm  growth  projections  and  a  discussion  of 
the  limits  to  continuing  growth  of  the  present  DDM  are 
presented  in  Section  3. 
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3.0  WHBRI  DDM  NBBD8  TO  00 

In  this  Section  %ie  discuss  the  forthcoming  growth  of  DDK  In 
terms  of  the  needs  of  the  user  community  end  the  limits  to  a 
DDM  using  currently  planned  trunking  and  PSMs.  The  material 
In  this  Section  draws  heavily  upon  Bolt  Beranek  and  Me%nMin's 
Future  Network  Technology  Study  (PNTS)  for  DCA  (the  Interim 
Report  Is  (HBR  841,  the  Pinal  Report  is  (HBR  85a 1,  and 
portions  of  the  report  are  summarised  In  (HBR  8Sb])  and  the 
Draft  DCA  White  Paper  On  DDM  Capacity  (WP)  (PRI  851. 

3.1  Future  DDM  Requirements 

Both  the  FMT8  and  the  WP  note  problems  In  forecasting  the 
groirth  of  DDM.  Hard  Information  to  support  such  a  forecast 
Is  not  available.  The  FMT8  produced  conclusions  that 
Indicated  that  the  msxlmnm  number  of  hosts  per  backbone 
network  could  rise  to  25,000  in  1988;  other  numbers  In  their 
reports  are  very  general.  Two  approaches  %iere  used  to 
obtain  estimates:  top-down  -  based  on  planned  computer 
Installations,  and  bottom-up  -  based  on  the  User 
Requirements  Data  Base  (URDB).  The  first  method  potentially 
overestimates  connections  to  DDM;  the  second  method 
potentially  underestimates  connections  to  DDM.  in  either 
case  there  are  unknown  factors  ^Ich  affect  the  actual 
number  of  computing  systems  %rhlch  are  candidates  for 
connection  to  DOM. 

The  WP  used  an  approach  which  examined  the  limiting  factors 
to  the  gro%ith  of  DDM  under  its  current  plans  (those  sum¬ 
marized  In  Section  2.4).  One  factor  Is  the  maximum  through¬ 
put  which  can  be  provided  by  the  trunking.  This  Is  based  on 
assumptions  that  the  topology  and  average  hop  length  will 
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reMln  about  the  saae.  Another  factor  Is  the  throughput  of 
the  switches.  Both  of  these  estlsAtes  result  In  a  saxlsnia 
nettrork  capacity  of  6  to  7  Mbps.  Trunking  Is  the  Halting 
factor  and  Halts  the  nuaber  of  hosts  to  about  2100  under 
the  assuaptlon  that  the  hosts  are  connected  by  9.6  kbps 
accsss  lines  idiich  are  30%  utilized. 

There  are  t%K>  user  conaunity  factors  which  will  have  a  aajor 
iapact  that  cannot  be  quantified  at  this  tiae.  The  first  is 
the  extent  to  which  Local  Area  Networks  (LANs)  will  keep 
local  traffic  off  of  the  backbone  network  and  concentrate 
distant  traffic  through  a  single  gateway  connection.  The 
second  is  the  changes  which  will  occur  in  the  types  of 
traffic;  this  reflects  the  difference  as  traffic  aakeup  goes 
froa  heavily  interactive  to  general  operational  use. 
Intelligent  terainals  and  PCs  will  greatly  reduce  the  nuaber 
of  single  saa 11-packet  aessages. 

There  are  also  current  logical  Haitations  to  addressing 
which  can,  in  principle.  Halt  growth.  These  Haitations 
can  be  overcoae.  The  Haitations  arise  in  AHXP  and  in  the 
DOM  Internet  Protocol  (IP)  ^ich  both  Halt  the  nuaber  of 
nodes  to  256  (for  technical  reasons  this  is  actually  253), 
and  the  nuaber  of  hosts  per  node  to  64  (AHIP)  or  256  (IP). 
These  Haitations  could  be  greatly  eased  if  the  DDN  X.25 
addressing  is  adopted  allowing  999  P8Ns  with  99  hosts  per 
P8M.  Adoption  of  the  180  Internet  Protocol  would  reaove  all 
IP  addressing  Halts. 

Over  and  above  the  question  of  addressing,  P8M  capacity 
Halts  are  discussed  in  both  the  rMT8  and  the  VP.  These 
Halts  deal  with  aeaory  Halts  for  tables,  overhead  for 
routing  updates  and  routing  coaputation  overhead,  and  packet 


14 


SPARTA,  INC. 


22  April  1986 


handling.  It  la  necessary  to  discriaiinate  between  three 
types  of  traffic: 

that  originating  or  terminating  at  a  P8N  and  going  to 
or  from  a  trunk 

that  going  to  or  from  another  host  connected  to  the 
same  P8M 

that  passing  from  one  trunk  to  another  through  a  PSN 
(tandem  traffic 

Both  reports  conclude  that  a  uniform  network  of  253  c/300 
P8Ms  could  support  approximately  7000  hosts  «fhlch  ate  con¬ 
nected  to  the  pans,  on  the  average,  by  30%  utilised,  9.6 
kbps  access  lines. 

3.2  Mew  Technologies  in  DoD 

The  advances  and  spread  of  computing  technology  has  a  poten¬ 
tial  Impact  on  DOM.  There  are  now  many  users  of  personal 
computers  (PCs)  who  have  the  potential  desire  or  need  to 
access  resources  across  a  data  coamunlcatlon  network.  These 
PCs  are  no  longer  the  "dumb”  terminals  which  are 
accoaaK>dated  by  TACs.  They  could  perform  the  functions  of 
hosts;  however,  DOM  can  not  be  expected  to  provide  host 
status  to  each  Individual' -PC.  PC's  can  be  expected  to  exist 
on  Local  Area  Metworks  (LANs)  within  appropriately  grouped 
physical  complexes.  Bach  LAM  would  liave  a  gateway  to 
connect  It  to  DDM;  the  gateway  appears  as  a  host  to  the 
MGP8.  Any  transaction  between  a  PC  and  another  PC  or  a  host 
on  the  DDM  would  be  accomplished  by  passing  traffic  through 
the  gateway  and  across  the  nettrork  to  the  appropriate  host 
or  gateway  on  the  LAM  for  the  other  PC. 
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3.3  Mew  Application  Level  Requireaents 

The  spread  of  coaputlng  resources  will  sncourega  ths  lapis- 
sMntatlon  of  distributed  applications  for  users  who  are 
adalnlstratlvely  related,  although  not  physically 
collocated.  These  users  will  per fora  file  transfers  of 
relatively  large  voluass  of  data  representing  one  of  the 
contributing  factors  to  the  gro%rth  of  user  regulreasnts  for 
DDM. 

At  the  present  tlas,  a  aajor  source  of  short  aessages  Is 
traffic  froa  duab  teralnals  to  hosts;  In  auch  of  this  the 
actual  data  per  single  packet  asssage  Is  a  single  byte 
(character).  This  traffic  occurs  when  a  teralnal  Is 
carrying  out  an  Interactive  application  on  a  reaote  host. 
It  represents  a  aajor  Inefficiency  In  network  usage.  As 
aore  Intelligent  teralnals  (l.e.  PCs)  coas  Into  use  an 
effort  needs  to  be  aade  to  deal  with  these  transactions  at 
the  level  of  screen  lines,  further,  we  can  expect  that  soas 
of  the  applications  aay  be  accoapllshed  on/at  the 
Intelligent  teralnal. 

As  users  gain  access  to  the  net  froa  their  '*own”  coaputers, 
new  applications  trhlch  generate  relatively  large  voluass  of 
traffic  such  as  anil  will  spread  rapidly;  however,  the  exact 
effects  of  application  changes  are  unknown  at  this  tlsa. 
Coaputer  Interaction,  like  coaputer  utilisation,  seeas  to 
obey  a  Parkinson's  law  of  expanding  to  utilise  all  of  the 
available  resources.  Ws  have  chosen  to  use  the  prediction 
that  average  host  traffic  will  aultlply  by  4  for  purposes  of 
slslng  the  MCPR  network  envlronasnt  as  used  In  Section  3.6 
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3.4  Security  Considerations 

Current  DOM  policy  [DON  85,  SHI  86,  LAN  85]  calls  for  sepa¬ 
rate  8ubnet%n>rks  for  each  security  level  with  the  subnet- 
iforks  thesMelves,  Including  the  PSHs,  being  operated  at 
systea  high  for  that  security  level.  This  situation  will  be 
aaellorated  when  BLACKER  provides  B3  for  aulti-level 
security  bet%ieen  end  users. 

There  is  also  a  requlreaent  that  network  control  ■essages  be 
authenticated.  This  can  best  be  acconpllshed  by  using  a 
cryptographically  based  checksum.  The  Bsssages  should  also 
be  protected  by  link  encryption  tdille  on  the  trunks. 

The  general  requirements  for  network  security  are  still  In 
the  process  of  evolution.  The  National  Computer  Security 
Center  has  a  draft  "Department  of  Defense  Trusted  Network 
Bvaluatlon  Criteria”  (DoDTNBC)  CC8C  85]  at  the  present  time; 
the  date  of  completion  of  DoDTNBC  is  uncertain.  This 
document  Is  Intended  to  be  analogous  to  the  "Department  of 
Defense  Trusted  Computer  Bvaluatlon  Criteria”  (DoDTCBC) 
(CSC  83],  the  so-called  "orange  book”.  DoDTCBC  provides 
standards  for  assessing  uniprocessors  and  their  operating 
systems.  Current  DON  plans  call  for  the  computers  within 
PSNs  to  meet  the  C2  criteria,  or  better,  of  DoDTCBC.  This 
requirement  Is  moderately  stringent.  Further,  application 
software  for  PSNs  should  be  written  with  computer  security 
validation  as  a  goal. 
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3 . 5  Other  Peaturee 

3.5.1  Precedence 

The  DON  requires  that  there  be  a  precedence  and  preeaption 
capability  within  DON  to  allow  for  efficient  response  to 
critical  users  while  the  network  is  under  various  levels  of 
stress  conditions  (p  129  of  the  DON  Proqraa  Plan  CDDM  821). 
Precedence  inforaation  anst  be  acted  upon  by  the  P8M  soft¬ 
ware  to  provide  the  required  service. 

3.5.2  Pairness 

Pairness  iaplies  that  all  users  of  equal  status  in  teras  of 
precedence  and  preemption  deserve  an  equal  chance  at  network 
resources.  lapleasntation  of  conqestion  control  can,  if 
done  in  a  siaplistic  %My,  deny  service  to  users  in  an  unfair 
manner.  There  are  proposals  within  the  data  communication 
community  and  within  the  plan  for  conqestion  control 
experiments  to  provide  fair  service  to  users,  all  other 
factors  being  equal.  This  is  a  desirable  feature  for  DDM. 

3.5.3  Type  of  Service 

Type  of  service  routing  can  improve  the  performance  of  a 
network  with  heterogeneous  trunking;  the  situation  «#e 
postulate  for  the  MOPS  environment  in  Section  5.  High  band¬ 
width  trunks  with  moderate  delay  are  very  appropriate  for 
file  transfers  and  for  mail.  Low  delay  trunks  can  be  used 
preferentially  for  short  msssages  and  Interactive 
applications. 
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3.6  The  MOPS  Network 

The  Halt  o£  7000  to  8000  hosts  discussed  in  Section  3.1  Is 
unrealistic  Insofar  as  It  assuaes  an  even  distribution  of 
hosts  to  PSMs  and  even  average  usage  by  the  hosts  at  a  given 
PSN.  Thus,  it  is  an  upper  bound  to  the  nuaber  of  hosts  that 
can  be  supported  by  the  straightforward  evolution/expansion 
of  the  current  technology. 


If  %re  use  the  projection  of  hosts  in  the  PMTS,  we  can 
envision  for  1990  a  network  which  has  froa  2000  to  30,000 
hosts.  At  the  low  end,  this  load  could  be  accoMMdated  by  a 
subset  of  their  projection  of  the  7000  host  network  of  253 
C/300S.  We  can  presuaw  this  to  be  an  unevenly  populated 
network  with  P8Ms  being  either  a  alx  of  C/300s  and  C/30ls  or 
all  C/30s.  At  the  high  end,  the  requlreaents  clearly  exceed 
the  projected  capacity.  In  any  event,  «fe  can  expect  that 
there  will  be  a  tlaa,  as  discussed  In  the  next  section,  in 
which  the  C/3x  based  network  with  64  kbps  trunks  will  becoas 
Inadequate . 


We  can  postulate  an  Idealised  network  which  supports  20,000 
hosts  as  follows: 


Assuae  a  20,000  host  network  with  400  nodes  and,  an 
average  of  SO  hosts  per  node,  and  an  average  hop  length 
of  4.  In  keeping  with  the  discussion  of  Section  3.3 
above,  we  shall  assuae  that  the  average  host  generates 
4  tlass  the  traffic  of  current  hosts,  30%  loading  on  a 
9.6  kbps  line  with  750  bit  packets,  or  alaost  4  packets 
per  second.  Based  on  this  nuaber  we  find  that  the 
average  traffic  entering  and  leaving  a  node  Is  0.6  Kbps 
(16  packets  tiaes  50  hosts  or  800  750-blt  packets  per 
second).  If  we  now  assuae  that  20%  of  this  traffic  Is 
Intranode  (the  current  value)  then  80%  of  the  traffic 
will  be  going  to  the  trunks.  With  our  average  hop 
length  of  4,  another  320%  of  the  0.6  Mbps  aust  be 
accosao dated  as  tandea  traffic.  Thus,  the  net  trunk 
capacity  of  the  MOPS  aust  be  2.0  Mbps.  This  Increase 
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In  capacity  o£  a  PSN  calls  for  the  total  throughput  for 
the  three  types  of  traffic  through  the  switch  to  be  as 
follows : 

Tvpo  of  Traffic  Throughout  (oackets/aec) 

Host  -  Trunk  640 

Trunk  -  Trunk  2,560 

Host  -  Host  160 

The  result  of  this  exaBg>le  calls  for  the  HOPS  to  have  an 
aggregate  throughput  of  3360  packets  per  second;  this  Is 
about  6  tiaes  the  capacity  of  the  C/300  (p  44  of  the  VP) 
The  significance  of  this  requlreasnt  is  discussed  In 
Appendix  A.  The  MOPS  requires  about  2  Mbps  of  trunk 
capacity.  Sohm  of  these  trunks  say  have  significantly  higher 
rates  than  64  kbps.  For  an  average  transit  length  of  4  hops 
an  evenly  distributed  net%#ork  should  have  5  trunk 
connections  per  node.  In  order  to  allow  for  uneven 
distributions  of  connectivity  and  traffic,  %re  propose  that 
the  saxisMl  switch  have  provision  for  aore  than  three  tiaes 
this  nuaber,  i.e.  20  trunk  connections. 

Section  6  presents  requlreaents  for  the  MOPS  based  upon  the 
above  aaterlal  (Sections  3.1  through  3.6). 

3.7  The  Date  at  Which  the  MOPS  is  Required 

we  will  aake  two  estiaates:  one  is  the  date  by  which  a 
network  eaploying  C/30ls  and  C/300s  as  PSMs  will  no  longer 
suffice;  the  second  is  the  date  by  which  a  network  of  C/30ls 
will  no  longer  suffice.  The  first  estiaate,  based  on  dis¬ 
cussion  sarlier  in  Section  3,  will  be  the  date  at  which  the 
nuaber  of  hosts  being  served  exceeds  about  7000.  If  we  take 
the  aiddle  ground  of  the  five  scenarios  of  the  PMT8  Inter ia 
Report  (p  46  (HBR  841),  this  date  will  be  about  the  aiddle 
of  1990. 
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Fox  auklng  the  second  estiaate  %re  will  assuae  a  network  of 
C/30B8  with  aoa«  doubling  up  at  high  throughput  nodes.  This 
suggests  that  a  4000  host  network  will  begin  to  exceed  the 
capacity  of  the  C/30B  based  network.  Again,  taking  the 
Middle  ground  froa  the  FMT8  Interla  report  find  a  date 
for  4000  hosts  to  be  about  the  Middle  of  1987. 

Using  the  first  estlaate  %ie  propose  that  a  date  of  January 
1991  for  fielding  the  first  NOPSs  would  be  satisfactory. 
Such  a  date  would  allow  sufficient  tlae  for  a  full 
developaent  cycle  for  an  all  new  MOPS  If  that  Is  selected  as 
the  saans  of  acquisition  (see  Section  8).  Using  the  second 
estlaate  %fe  propose  that  a  date  of  January  1988  for  fielding 
the  first  MOPS  would  be  satisfactory.  This  would  not  give 
tlae  for  a  full  developaent  cycle;  It  could  accosaadate  the 
strategy  of  Modifying  a  coaaerclal  switch  to  be  the  MOPS 
(see  Section  8).  Schedules  for  both  cases  are  presented  in 
Section  9. 
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4.0  WHIRI  TICHMOLOGY  IS  OOIMO 

Section  3  showed  that  the  DM  will  need  to  handle  both  a 
greater  nuaber  oC  hosts  and  a  auch  larger  voluae  of  traffic. 
This  growth  is  one  of  the  aajor  drivers  for  the  Next  Genera¬ 
tion  Packet  Switch  (MGPS).  In  addition,  technological 
developaents  are  occurring  «diich  will  have  an  iapact  on  the 
NGPS.  Discussions  of  these  developaents  will  be  found  in 
(HBR  84,  HBR  85a,  ANA  84,  PRA  76,  and  PRA  791.  Increased 
use  of  computing  resources  will  be  the  result  of  widespread 
access  to  computers,  other  technological  developaents  will 
support  both  expansion  and  grotfth;  they  will  also  lead  to 
changes  in  the  net%rork  and  the  switches.  Changes  in  trunk 
and  access  line  transmission  technologies,  in  computing 
technology,  and  in  coaswrcial  communication  systems 
architecture,  together  with  special  DM  and  DoD  requirements 
will  all  contribute  to  the  requirements  for  the  MOPS.  These 
are  discussed  below. 

4.1  Transmission  Technology 

Packet  switched  net«rorks  will  have  a  wider  range  of  tech¬ 
nologies  for  the  consranication  links  with  both  higher 
bandwidths  and  a  mix  of  delays  available.  Teleconmunlca- 
tions  transmission  technology  is  in  a  period  of  rapid 
change.  Por  high  bandwidth  transmission,  microwave  and 
coaxial  cable  transmission  systems  have  been  overtaken  to  a 
considerable  extent  by  satellite  and  more  recently  fiber 
optics  development.  Por  lower  data  rates,  improvesmnts  in 
modems  make  rates  up  to  19.2  kbps  readily  achievable  over 
classical  voice  grade  lines*  At  the  same  tine,  voice 
transmission  techniques  are  becoming  all  digital  as 
discussed  in  Section  4.3  below.  All  of  these  developments 
will  have  an  impact  on  the  future  DM  and  the  MGPS. 
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4.1.1  Satellite  TransAieelon 

Satellite  transaission  affects  a  packet  switched  netirork  in 
tiro  %#ays.  The  delay  over  a  single  hop,  assnalng  a  geosynch¬ 
ronous  satellite  is  significant,  it  is  on  the  order  of  half 
a  second.  This  delay  has  an  iMsdlate  effect  on  the  window 
size,  the  nusdrer  of  Messages  which  are  allowed  to  be  In 
flight  without  an  acknowledgesMnt  being  received.  The  old 
network  protocols  provided  for  a  MazlMusi  window  size  of  8; 
the  new  n  protocol  changes  that  to  128.  In  addition  anch 
higher  bandwidth/ increased  data  rates  are  available.  This 
higher  bandwidth  can  be  exploited  In  either  of  two  ways;  It 
can  be  utilised  as  a  nua8>er  of  sMiltlplexed  low-speed  chan¬ 
nels  or  as  one,  or  a  few,  high-speed  channels.  Satellite 
channels  norsally  Incorporate  sobm  "built-in”  error  correc¬ 
tion  and  thus  are  aore  reliable  than  the  original  channels 
used  on  ARPAHtT.  As  a  result,  packet  sizes  can  be  larger 
than  those  used  for  ARPAHIT  and  a  packet  size  of  8000  bits 
would  iaprove  use  of  satellite  bandwidth  CMIL  811.  Larger 
packets  are  also  aore  effective  when  transactions  are 
Insensitive  to  satellite  delays. 

4.1.2  High  Speed  Terrestrial  Transalsslon 

High  speed  terrestrial  transalsslon  is  growing  and  becoalng 
less  expensive  as  a  result  of  the  growth  of  fiber  optics 
systesa  and  the  related  growth  of  T1  carrier  services. 
Bandwidth  and  error  rate  will  be  slallar  to  that  discussed 
above  for  satellites.  Conaerclally  In  North  Aasrica  fiber 
optics  transalsslon  systeas  will  be  divided  Into  T1  carriers 
their  Multiples,  and  subaultlples.  (Per  exaaple,  384  kbps 
will  be  a  coaaon  subset  of  both  T1  and  the  Buropean  2.048 
Mbps  lines).  Again,  because  of  the  loiier  error  rate  longer 
packets  are  practical  If  the  data  transalsslon  needs  can 
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support  them.  *  The  lo%«er  delay  tlae  represents  an  iaprove- 
Mnt  over  satellite  channels  for  interactive  applications. 

Another  role  of  fiber  optics  is  its  forthcoainq  use  in 
undersea  cable  systesM.  This  usage  ellainates  aost  of  the 
delay  occurring  in  satellite  systeas  referred  above;  a 
trans-Atlantic  delay  «iould  be  less  than  10  nsec. 

4.2  Conputing  Technology 

Conputing  technology  is  both  the  driving  force  in  the  expan¬ 
sion  of  conputer  use  and  the  support  for  packet  switches. 

This  technology  continues  to  progress  rapidly;  it  iaproves 
by  a  factor  of  2  in  coat  per  unit  of  conputation  every  two 
years.  Inportant  areas  for  the  MOPS,  architecture  and 
engineering,  devices,  and  conputer  security,  are  discussed 
below. 


4.2.1  Architecture  and  ingineering 

Developnent  of  conputing  systeas  with  vary  high  conputatio- 
nal  throughput  at  ainiaal  cost  is  a  current  najor  trend.  As 
speed  boundaries  are  approached  (e.g.  1  nanosecond  for  a 
electrical  inpulse  to  traverse  about  7*  of  wire),  it  is  no 
longer  econoalcal  to  force  higher  throughput  by  sheer  serial 
speed  up.  As  a  result,  especially  where  problens  can  be 
partitioned  into  tasks  which  can  run  concurrently  with 
little  or  no  inter-task  coaannication  required,  parallel  or 
concurrent  processing  becoaws  very  attractive. 

Many  new  conputer  architectures  are  providing  nnltiple  pro¬ 
cessors  for  parallel  processing.  Vith  the  advent  of  LSI  and 
VLSI,  assenblies  of  identical  processors  becons  econoaically 
attractive  as  well.  These  anltiprocessors  have  the 
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potential  to  provide  redundant,  tall-soft  configurations. 
Although  not  all  problesM  can  be  partitioned  Into  parallel 
tasks,  packet  switch  reguiresMnts  are  well  suited  to  such 
partitioning.  On  the  other  hand,  operating  systesM  for 
mltiprocessors  are  not  yet  at  a  asture  stage. 

Hardware  and  Software  Inglneering,  particularly  with  compu¬ 
ter  aids,  are  aaking  steady  advances.  Thus,  it  is  possible 
to  build  systesM  of  chip  level  devices  in  a  relatively 
straightforward  aanner.  Similarly,  software  applications 
can  be  built  in  an  orderly,  modular,  and  self -documenting 
way  which  is  easy  to  understand,  make  operational,  and 
modify. 

4.2.2  Logic  and  Memory 

LSI  and  VLSI  technology  continue  to  mske  advances  by  factors 
of  about  two  every  2  years.  These  advances  can  be  In  any  of 
three  dimensions:  higher  complexity/s ise;  higher  speed;  or 
lower  cost.  As  a  result,  devices  such  as  microprocessors, 
memories,  and  specialized  microcontrollers  which  can  be 
produced  on  a  single  chip  attain  great  advantage.  Memory 
size,  especially,  is  no  longer  a  cost  problem  and  error- 
correcting  memory  systems  produce  more  reliable  memories 
than  ever  before.  for  packet  switches,  specialized  micro¬ 
controllers  include  comsNinication  controllers  and  hardware 
implementations  of  cryptographically  based  or  other  checksum 
algorithms. 

4.2.3  Computer  Security 

Computer  security  is  another  area  which  is  receiving  signif¬ 
icant  attention  at  this  time.  The  National  Computer 
Security  Center  has  developed  criteria  for  trusted  computer 
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systeas  (CSC  831  and  is  developing  criteria  for  trusted 
netirarks  (CSC  851.  These  activities  are  beginning  to  influ¬ 
ence  the  cosBMrcial  coaputer  environaent  although  auch 
reaains  to  be  done. 

4.3  ConsMrcial  CoasMinication  Technology 

Analog  telecoaannications  are  becoaing  obsolete.  The  "tele¬ 
phone  industry*  is  rapidly  aoving  toMrd  providing  a  digital 
technology  based  service.  As  a  result,  new  digital  connec¬ 
tions  to  subscribers  are  becoaing  available  and  there  is  a 
strong  iapetus  to  provide  both  voice  and  packet  switched 
data  service  using  as  auch  coaaK>n  plant  as  possible.  Soae 
of  this  is  exeaplified  by  the  dsvelopaents  in  the  Integrated 
Services  Digital  Network  (ISOM)  and  in  the  developaent  of 
switches  that  provids  for  both  circuit  and  packst  switching. 
These  two  topics  are  discussed  below. 

4.3.1  integrated  Service  Digital  Network  (ISDN) 

The  ISDN  is  an  architecture  and  a  philosophy  that  is  cur¬ 
rently  being  developed  at  the  standards  level  by  the  CCITT. 
The  official  description  of  X8INI  is  in  the  I-Series  of  CCITT 
recoasMndations  ICC!  651;  NITM  has  prepared  a  set  of  papers 
for  DCA  discussing  ISDN  and  its  potential  iapact  on  DCA 
(SAK  83a,  SAK  83b,  8«S  83a,  SMB  83bl.  IBBB  Cossunications 
Magasine  has  also  devoted  a  special  issue  to  ISDN  (XIB  861. 
ISDN  calls  for  ths  use  of  all  digital  techniques  for  switch¬ 
ing,  any  rsquisits  intarnediate  storage,  and  transnission 
froa  one  subscriber's  devics  to  that  of  another  subscriber. 
ISDN  will  use  a  contention-avoidance  local  bus  for  subscri¬ 
ber  premises  and  digital  (i.e.  cosputer  or  cosputer  tech¬ 
nology  based)  interfaces  and  switches  throughout  the  overall 
ISDN.  This  loop  will  have  ths  "2S4D”  capacity,  2  B  cliannels 
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at  a  64  kbps  rate  and  one  D  channel  at  a  16  kbps  rate.  The 
B  channels  were  nornally  Intended  to  be  used  for  digitized 
voice  and  the  D  channel  was  to  be  used  for  data  and 
signalling  for  all  channels.  The  B  channels  can,  in  fact, 
also  be  used  for  data. 

A  najor  difference  between  ISDN  and  auch  of  the  current 
teleconaNinications  technology  is  the  use  of  *coaaK>n  channel 
signaling*.  That  is,  for  circuit  switched  applications,  the 
control  Infornation  (call  set-up,  etc.)  is  transaitted  on  a 
separate  all  digital  channel.  Extensive  use  is  to  be  aade 
of  high  bandwidth  channels  (and  their  aajor  subdivisions) 
such  as  North  Aaerican  T1  at  1.544  Mbps  or  the  siailar 
Buropean  service  of  2.048  Mbps.  ISDN  for  T1  will  noraally 
be  "23-B40”,  23  B  channels  and  one  D  channel  for  signalling. 
The  B  channels  can  be  aade  availble  to  a  switch  in  a  flex¬ 
ible  way)  the  D  channel  will  be  used  for  signalling  and 
adainistration.  An  extension  to  the  18DN  Standards  is  under 
way  to  specify  how  packets  are  sent  over  B  channels,  using 
the  LAP  B  protocol,  and  how  the  0  channel  will  be  used  for 
high  level  control. 

The  advent  of  ISDN  and  its  supporting  transaission  tech¬ 
nology  will  increase  the  availability  and  reduce  the  cost  of 
high  bandwidth  cdsaunication  links. 

4.3.2  Hybrid  Switches 

Although  there  has  been  extensive  experiaentation  with 
packet  voice,  digital  voice  which  is  further  segaented  and 
sent  over  a  packet  switching  systea,  there  are  a  nuaber  of 
drawbacks  to  its  widespread  use.  Two-way  voice  cossMinica- 
tion  is  not  tolerant  of  highly  variable  delay;  this  aili- 
tates  against  error  correction  by  requesting  re transaission. 
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In  fact,  Most  digitized  voice  (say,  at  9.6  kbps  or  above)  is 
tolerant  of  occasional  errors  and  does  not  require  error 
correction.  Finally,  although  speech  Is  also  "bursty".  It 
is  desirable  to  handle  full  "talkspurts*  which  are  a  second 
or  »ore  In  length.  Such  talkspurts  represent  froa  tens  to 
hundreds  of  packets,  depending  upon  the  voice  coding  tech¬ 
niques.  There  ate  proposals  to  do  a  fora  of  fast  circuit 
switching  for  talkspurts;  this  Is  called  "burst  switching" 
(HAU  83). 

As  a  result  of  these  voice/data  distinctions,  within  the 
cosswrclal  telecoaaunlcatlons  area  there  has  been  Increasing 
attention  paid  to  the  concept  of  "hybrid  switches".  These 
are  coaputer  based  switches  which  provide  both  circuit 
switching  for  digital  voice  and  packet  switching  for  data. 
For  exaaple,  Budrlkis  and  Metravall  have  proposed  a  design 
for  a  hybrid  switch  IBtIO  84)  In  which  circuit  switching  Is 
provided  by  repeatedly  allocating  asaory/  transalsslon  slots 
for  a  given  call  and  packet  switching  Is  achieved  by 
coapetltlon  for  the  unused  slots.  The  AT8T  5B88  described 
In  8ectlon  7.1  below  represents  another  hybrid  switch. 

4.4  8uanary  and  Conclusions 

The  current,  rapidly  changing,  technology  Is  both  a  driver 
for  DON  growth  while  at  the  saae  tlas  It  can  provide  support 
for  a  anch  larger  network  with  higher  traffic  voluaes  for 
users.  The  requlreaents  for  the  NOF8  developed  In  8ectlon  6 
are  coapletely  in  accord  with  the  current  and  very  near  tera 
technology. 
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5.0  THB  HOPS  BMVXRONHBHT 

5.1  Introduction 

In  Section  1.2  we  Mntloned  that  this  study  has  concluded 
that  the  most  likely  network  environment  for  the  NOPS  Is  an 
extension  of  the  present  network  topology  but  that  greater 
throughput  la  needed  for  both  PSMs  and  for  trunks.  More 
flexibility  will  also  need  to  be  provided  as  discussed  in 
section  3.  These  ideas  are  elaborated  in  Section  5.2  below. 
Some  of  the  possible,  but  "rejected"  environments  are 
discussed  in  Appendix  B  in  more  detail;  we  have  concluded 
that  they  are  unlikely  environments  in  the  anticipated  time 
frame  for  the  MOPS. 

5.2  Global  Strategy 

An  extension  of  the  current  topology  and  connectivity 
appears  to  be  desirable  for  two  primary  reasons: 

the  network  can  continue  to  evolve  without  drastic 
changes , 

no  compelling  reasons  appeared  to  make  the  other 
alternatives  stand  out  as  attractive. 

This  conclusion  is  also  one  that  is  expressed  in  the  Draft 
White  Paper  on  DON  Capacity  (PRI  851  which  calls  for  modular 
evolutionary  growth  without  major  disruptions. 

We  note  here  that  one  of  the  more  interesting  possibilities 
presented  in  Appendix  B  is  that  of  a  hybrid  switch  which 
shares  trunking  facilities  with  a  voice  circuit  switch. 
Such  an  arrangement  allo%rs  for  maximum  flexibility  in  using 
DoD  owned  or  leased  transmission  resources;  it  Is  also  tech¬ 
nologically  in  line  with  current  voice/data  integration  as 
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exemplified  by  ISDN  and  dlscusaed  In  section  4.3.  The  use 

of  hybrid  switches  requires  investigation  by,  and 
coordination  with,  the  voice  planning  process  of  OCA. 

As  a  result,  %re  believe  that  the  NOP8  environment  will  be 
that  characterized  in  Figure  5-1.  This  is  an  illustrative 
diagram  %fhich  indicates  the  kind  of  connectivity  expected 
between  nodes.  The  intent  Is  to  show  something  similar  to 
the  present  DON.  Three  sajor  differences  from  the  current 
DDN  are  expected: 

There  will  be  differing  types  of  trunks  between  a  given 
pair  of  nodes  (for  example,  trunks  of  differing  speeds 
and  delays). 

Trunk  speeds  significantly  greater  than  64  kbps  will 
need  to  be  accommodated, 

A  greater  number  of  hosts  will  need  to  be  accommodated 
at  some  high  traffic  density  nodes. 

we  believe  that  these  differences  are  justified  by  the 
extrapolations  developed  in  Section  3.  Further,  the  HOPS 
will  need  to  be  able  to  have  most  of  the  new  functionality 
which  is  being  or  will  be  placed  into  service  for  DDN. 
These  are  such  features  as  the  new  end-to-end  protocol, 
congestion  control,  and  type  of  service  routing.  It  will 
also  have  to  have  the  requisite  security  features  and  such 
DoD  features  as  precedence  and  priority.  The  requirements 
developed  In  Section  6  and  summarized  In  the 
recommendations  of  Section  10  are  based  on  this  environment. 
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S.3  Switch  Families 

In  Section  8  there  is  a  discussion  o£  two  %fays  of  acquiring 
the  MOPS:  development  of  a  new,  DDM  specific,  switch;  or  use 
of  a  switch  based  on  conamrcial  developments.  Many  of  the 
commercial  switches  described  in  Section  7  are  available  in 
"families"  which  provide  the  same  service  but  with  different 
throughputs,  number  of  ports,  etc.  The  MOPS  can  also  be 
usefully  provided  as  a  family  with  a  range  of  capacities. 
Since  we  will  have  a  range  of  load  requirements  for  PSNs,  a 
family  of  MOPSs  will  provide  a  more  cost  effective  fit  to 
those  conditions. 
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6.0  NOPS  REQUIRBHBNTS  DISCUSSION 

This  Report,  and  particularly  this  Section,  develop  a  pro¬ 
posed  set  of  requiresMnts  for  the  NOPS.  These  NGPS  require¬ 
ments  will  have  to  be  interpreted  in  the  light  of  an  overall 
DON  program  plan  which  will  be  aimed  at  meeting  the  service 
requirements  of  the  1990-1995  time  frame.  These  service 
requirements  have  been  estimated  in  Section  3  and  expect  the 
DDN  program  plan  to  be  in  general  accordance  with  the 
discussion  of  the  NOPS  environment  in  Section  5.  Some  of 
the  requirements  presented  here  may  have  to  be  waived  or 
modified  (for  example.  Section  6.2,  Implementation,  below) 
If  a  commercial  offering  is  adopted  or  leased. 

6.1  Switch  Specifications 

6.1.1  Quantity 

Following  the  argument  of  Section  3  we  estimate  that  HILNBT, 
the  largest  component  of  the  DDN,  will  require  about  400 
NOPSs.  This  suggests  that  a  maximum  of  1000  N(3PSs  will 
suffice  for  DoD/DCA  generally.  Given  the  state  of  hardware 
and  software  engineering  this  quantity  is  a  satisfactory 
size  base  for  developing  a  unique  architecture.  As  dis¬ 
cussed  in  Section  5.2,  the  architecture  should  allow  for  a 
family  of  switches;  that  is  the  NOPS  needs  to  be  modular  in 
its  hardware  design. 

6.1.2  Capacity 

The  throughput  performance  of  the  maximal  switch  can  be 
specified  by  the  requirements  developed  in  Section  3.6. 


When  specifying  a  feeily  of  switches^  other  eenbers  of  the 
family  could  be  specified  at  soas  fraction  (e.g.  3/4,  1/2, 
and  1/4)  of  this  maxieue  capacity. 

6.1.3  Interfaces 

Trunk  interfaces  should  be  provided  at  sultiple  data  rates; 
these  should  be  standard  line  rates  (e.g.  9.6  and  19.2  kbps, 
and  perhaps  56  kbps)  below  64  kbps.  64  kbps  trunks  should 
be  accomodated.  A  question  arises  regarding  trunk  rates 
above  64  kbps.  There  are  standard,  inexpensive,  and 
reliable  %#ays  of  using  a  T1  as  many  64  kbps  channels.  Using 
a  T1  or  a  major  subdivision  of  a  T1  as  a  single  high  speed 
trunk  will  require  new  transmitters  and  receivers  at  the 
electrical  level.  At  this  time  we  propose  that  the  external 
interfaces  to  trunks  be  modular  and  accomodate  64  kbps 
trunks;  provisions  should  be  made  to  substitute  faster 
interfaces  up  to  at  least  394  kbps.  That  is,  internal  to 
the  P8M,  384  kbps  rates  to  and  from  a  trunk  interface  can  be 
supported.  The  interface  protocols  should  be  DOM  X.2S. 

The  access  line  interfaces  should  acconswdate  line  speeds  up 
to  64  kbps;  this  will  allow  for  use  of  the  ISDN  B  channels 
as  access  lines.  Access  line  speeds  below  64  kbps  should  be 
standard  multiples  of  1.2  kbps.  AHIP  will  no  longer  be 
supported  (except  for  transitioning  -  see  Section  6.9 
below).  DON  X.25  should  be  adopted  for  host  access  lines. 
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6.2  lapleMentatlon 

The  discoeeion  of  lapleaentatlon  presented  here  Is  predi¬ 
cated  on  the  assusiptlon  that  a  new  HOPS  will  be  developed 
under  contract  to  DCA  .  Therefore,  acquisition  will  proceed 
through  the  usual  phases  and  appropriate  regulations 
concerning  developsMnt  of  coaputer  systeas  will  have  to  be 
followed.  As  noted  at  the  beginning  of  Section  6,  soae  of 
these  requireaents  will  not  apply  if  coaaercial  packet 
switches  are  acquired  or  eaployed. 

6.2.1  Hardware 

The  IKIPS  should  be  both  TIMPBST  and  HII9  protected.  Except 
for  those  specifics  it  should  be  constructed  to  the  best 
coaaercial  standards.  The  MOPS  should  be  capable  of  grace¬ 
ful  degradation  under  conditions  of  hardware  failure  (see 
also  Reliability/Maintainability  in  Section  6.3  below). 
This  iaplies  that  the  MOPS  is  at  ainiaua  a  dual  processor 
systea  (as  discussed  in  Section  4).  A  preferred  physical 
iapleasntation  would  be  one  in  which  link  encryptors,  and 
any  level  1  and  2  interfaces,  such  as  aodeas,  are  housed 
within  the  saae  cabinetry  as  the  HOPS,  facilitating  the 
TEMPEST  and  HEMP  protection. 

6.2.2  Software 

Software  fox  the  HOPS  should  be  designed  and  iapleaented 
using  a  high  level  language.  This  should  be  done  for  all  of 
the  reasons  associated  with  good  software  engineering 
practice  including  siapiicity  of  change  and  ease  of  aain- 
tenance.  The  choice  of  the  higher  level  language  should  be 
aade  at  the  tias  at  which  the  foraal  (A-level)  MOPS 
specification  is  developed.  One  obvious  candidate  is  ADA. 
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At  the  present  tlM  there  are  soae  objections  to  ADA  based 
on  the  beliefs  that  ADA  Is  not  "Mture”  enough,  does  not 
generate  efficient  object  code,  and  Is  not  a  good  basis  for 
secure  systesM.  The  first  two  of  those  beliefs  9my  be 
alleviated  by  tlsM;  the  last  is  countered  by  arguaents 
presented  by  Anderson  (AMD  851  In  which  he  aakes  a  case  that 
ADA  Is  a  sound  basis  for  secure  prograsa. 

6.3  Rellabillty/Malntalnabllity 

The  MOPS  should  have  provision  for  "fall-soft*  operation. 
That  la,  the  failure  of  a  sajor  coaponent  such  as  a 
processor  aodule,  a  aeaory  aodule,  or  a  potter  supply  should 
not  render  the  NGP8  Inoperative,  although  It  would  reduce 
the  throughput  capacity.  Although  soae  aajor  coaponent 
failures  alght  deny  service  to  a  particular  host  or  the  use 
of  a  particular  trunk,  they  cannot  be  peraltted  to  bring  the 
whole  switch  do%m.  Software  within  the  MOPS  should  be  pro¬ 
vided  to  aonltor  the  switch  status  on  a  continuing  basis;  to 
report  aalntenance  requlreaents  which  can  be  satisfied  by 
replacing  aajor  aodules.  It  is  within  the  state-of-the-art 
for  a  coabination  of  hardware  and  software  to  disconnect 
faulty  aodules  and  connect  spare  aodules.  Such  reaote 
aalntenance  should  be  presented  as  an  option  when  asking  for 
bids. 

Overall,  It  should  be  possible  to  specify  that  the  MOPS 
shall  provide  99.995%  availability  for  at  least  reduced 
operation,  a  swan  tiws  between  (partial)  failures  of  1000 
hours,  and  a  asan  tlae  to  repair  of  0.5  hours 
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6.3.1  Sttzvivability 

DON  survivability  for  catastrophic  evonts  is  onsursd  by  the 
ovsrall  rodundancy  of  ths  notvorfc;  thsrs  are  no  spsclal 
survivability  requirsasnts  for  ths  MOPS. 

6.4  Sscurity 

The  switch  will  be  expected  to  Met  at  least  the  Class  C2 
criteria  of  the  DoD  Trusted  Coaputer  Security  Ivaluation 
Criteria  (CSC  831.  The  Class  C2  criteria  specify  that  the 
coaputer  systea  be  required  and  trusted  to  Mintain 
discretionary  access  control.  As  noted  in  Section  3.4,  it 
is  not  clear  at  this  tlM  how  the  Dob  Trusted  Network 
Bvaluation  Criteria  (the  draft  is  (CSC  851)  will  apply  to 
the  switch.  Further,  it  is  not  clear  exactly  how  either  set 
of  criteria  apply  to  a  aultiprocessor  switch.  Xn  any  even, 
the  switch  should  be  designed  and  docuMuted  using  a  forMl 
developMnt  Mthodology.  The  NCHP8  (and  the  new  Network 
control  Center  when  it  is  specified  and  developed)  should 
provide  for  authentication  of  control  traffic  on  the 
network.  There  will  be  a  large  nuaber  of  link  encryptors  at 
each  switch.  These  link  encryptors  should  be  integrated 
with  the  switch  as  closely  as  possible  to  ainiaixe  cabling 
and  space  for  the  switch. 


The  DON  is  continuing  to  inprove  security  over  the  %ihole 
network.  The  plan  is  described  in  the  draft  docuMnt 
Defense  Data  Network  Nvolution  of  Security  Services  IDDN  851 
These  security  plans,  som  of  which  are  already  in  being, 
will  place  requireMnts  on  the  network  and  the  NOPS. 
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6.5  Unattended  Operation 

The  MOPS  should  be  capable  of  unattended  operation  and  of 
perforaino  certain  configuration  changes  by  either  local  or 
Network  Control  Center  direction.  These  changes  would  be 
those  discussed  under  saintainability/reliability  above, 
those  changes  necessary  to  accoaoMdate  line  failures,  and 
those  changes  dealing  with  revisions  in  the  coaposltion 
( naass/addresses )  of  the  net.  Changes  initiated  by  the 
MOPS  will  be  reported  to  the  Network  Control  Center. 

€.6  Network  Control  and  Accounting 

At  the  tine  that  the  new  DON  progran  plan  Is  developed  it 
will  be  necessary  to  develop  a  set  of  requlrenents  for  a 
Network  Control  Center.  There  will,  of  course,  be  aultlple 
Centers  for  rellabillty/survlvabllity  considerations.  If 
detailed  accounting  is  to  be  used  (as  currently  proposed) 
for  charging  purposes  there  will  also  need  to  be  a  aethod  of 
collecting  infornation  to  provide  that  accounting.  Account¬ 
ing  Is  conaonly  perforned  at  the  Network  Control  Center  In 
coaswrclal  applications.  It  alght  be  desirable  that  these 
functions  not  be  conblned  In  ODM  In  order  to  leave  the 
Mettrork  Control  Center  with  aaxlaua  capacity  for  handling 
stress  situations.  More  detailed  discussion  of  these 
requireaents  Is  outside  the  scope  of  this  Report. 

6.7  Transitioning 

The  MOPS  will  have  to  be  brought  Into  service  in  a  phased 
aanner.  Nodes  containing  the  MOPf  will  have  to  serve  trunks 
consmnlcating  with  nodes  containing  C/3x's  and  serve  links 
conauni eating  with  local  hosts.  As  a  result,  when  a  node 
becoass  an  MCV8  node  there  will  be  a  nuaber  of  possible 
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strategies.  Three  possible  strategies  and  a  brief  discus¬ 
sion  of  their  pros  and  cons  follow: 

The  MOPS  can  have  a  Mode  in  which  it  eaulates  the  C/3x. 

The  HOPS  can  be  provided  with  software,  flrswere,  or  a 
Mixture  to  sMulate  the  C/3x  and  perMit  its  operation  as 
a  C/3x  within  the  existing  net.  At  a  tine  when  a 
grouping  of  closely  connected  MOPSs  exist,  that  group 
will  be  switched  over  to  the  new  prograas.  This 
strategy  laplles  that  the  transition  will  take  place  in 
large  steps. 


The  MOPS  can  cooperate  over  a  trunk  with  a  collocated 
C/3x,  sharing  functionality  during  a  gradual  switchover 
of  trunks  and  access  lines  between  the  two  switches. 

A  coaaun lea t Ions  channel  will  be  provided  between  the 
MOPS  and  the  C/3x  ;  "transition”  software  will  be 
written  for  both  the  MOPS  and  the  C/3x  to  provide  for 
Moveaent  of  packets,  control  Inforastlon,  routing 
inforsMtlon,  etc.  between  the  two  switches.  As  access 
lines  and  trunks  are  Moved  froM  the  C3/x  to  the  MOPS  or 
as  new  access  lines  and  new  trunks  are  Installed  on  the 
HOPS,  More  and  More  of  the  load  will  shift  froM  the 
C/3x  to  the  MOPS.  This  Is  a  More  gradual  strategy;  it 
requires  keeping  dual  switches  In  operation  for  a 
transition  period.  A  variation  of  this  strategy  would 
be  the  provision  of  a  duplicate  netmrk  having  both  old 
and  new  switches  at  the  nodes,  the  duplicate  networks 
would  be  Interconnected  at  Many  of  the  nodes  by 
gateways . 

The  MOPS  can  provide  non-optlMlzed  C/3x  functionality 
as  well  as  full  MOPS  functionality. 

The  MOPS  will  be  prograxMed  to  perfora  the  the  Mlnlaal 
set  of  functions  to  perait  working  with  users  and  the 
C/3x*s.  This  will  be  done  using  the  protocols  which  are 
current  at  the  tlae  of  Installation.  The  MOPS  will 
also  be  progranaed  to  per fora  the  new  functions  and 
protocols  specified  for  the  MOPS.  Perforaance  of  the 
”old”  protocols  need  not  be  at  aaxlaua  efficiency  since 
this  Is  only  needed  for  the  transition  period  and  the 
MOPS  will  have  greater  capability  than  the  C/3x  being 
replaced.  As  is  the  case  with  current  P8N  software 
releases,  the  Network  Control  Center  can  instruct  the 
node  as  to  which  release  applies  for  various 
operations.  The  Network  Control  Center  thus  is  in 
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control  of  the  evolvinq  use  of  the  NOPSs.  A  dual  node 
operation,  sinilar  to  that  used  by  ATat  in  the  1PS8 
(see  Section  7.1.2)  would  facilitate  transitioning. 

All  of  these  strategies  require  additional  prograsniing  be¬ 
yond  that  required  for  a  nettrork  using  only  the  HOPS.  The 
last  strategy  is  the  nost  straightforward  and  Is  reconnended 
as  a  requlrenant.  That  strategy  %n>uld  be  facilitated  if 
future  releases  of  C/3x  software  are  specified  and  designed 
in  higher  level  languages. 
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7.0  IXAMPLI8  OP  COMNIRCXAL  PACKET  SWITCH  CAPABILITIES 

In  this  section  we  examine  soae  of  the  consMrclal  packet 
switches  for  two  reasons:  they  represent  possible  Ideas 
which  should  be  considered;  they  represent  possible  bases 
for  packet  switches  to  be  procured.  The  discussion  of 
current  offerings  Is  not  an  exhaustive  survey;  it  represents 
InforsMtlon  that  was  conveniently  available.  With  regard  to 
future  offerings  only  Horthern  TelecoM  and  ATAT  Bell 
Laboratories  had  very  such  to  say. 

7.1  Current  Cosnerclal  Switches 

7.1.1  Aanet 

Aanet  wakes  the  Nucleus  6000,  a  wiltlprocessor  packet  switch 
using  up  to  10  Intel  00206 *s  connected  together  by  a 
proprietary  bus  technology.  A  Mxlwm  of  2S6  front  end 
processors  handle  up  to  1024  ports.  Trunk  speeds  of  up  to 
64  kbps  are  accoawodated;  X.25  protocol  Is  used.  Within  the 
switch,  duplicate  paths  can  be  provided  for  hardware 
redundancy  and  assurance  of  higher  availability.  A  network 
■anagewent  function  Is  provided  on  the  saae  hardware  base. 
Aanet  states  that  throughputs  of  up  to  1000  packets  (of 
unstated  size)  per  second  can  be  acconnodated  on  their 
largest  configuration. 

7.1.2  AT6T 

ATAT  currently  has  their  No.  1  Packet  Switching  Systen 
(IPSS)  In  operation.  Like  their  No.  5  Electronic  Switching 
Systen  (5ESS)  which  is  described  in  ICAR  051,  the  1P88  Is 
based  on  the  ATT  3B200  (BSC  031  coaputer,  a  dual  processor 
used  for  the  prinary  control  and  supervisory  functions.  The 
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dual  procassors  of  the  3B20D  are  not  used  as  anultiprocessor 
systea;  they  are  the  control  processor  and  a  hot  standby. 
When  installing  a  new  software  release  one  processor  laple- 
sants  the  old  release  and  one  the  new  release  to  facilitate 
debugging.  In  the  overall  switch.  Individual  Microproces¬ 
sors  handle  8  ports  (access  lines  or  trunks)  each;  up  to  60 
of  these  Microprocessors  can  be  used  providing  for  a  total 
of  480  ports.  In  Release  3  of  the  1P88  all  Msssage  data  is 
passed  through  the  Main  MOMory  of  the  3B200s  using  DMA 
techniques.  The  capacity  of  the  switch  is  quoted  at  1200 
packets  per  second.  This  is  realistically  500  or  600  true 
data  packets  of  128  Bytes  each;  the  other  packets  are  used 
for  adMinistration  and  acknowiedgoMents .  Release  4  of  the 
1P88  (apparently  early  in  1987)  will  provide  inter processor 
coMMunlcation  via  a  32  Mbit  duplex  ring.  Up  to  960  ports 
can  be  accoModated  and  the  throughput  is  to  be  over  4000 
packets  per  second. 

The  1P88,  using  additional  software,  can  operate  as  a 
Nat«rork  Control  Center  8ysteM  (MCC8).  The  switches  are 
intended  for  unattendad  operation  and  extensive  centralised 
diagnostics  and  display  ace  provided;  these  can  be  run  at 
any  1P88.  Up  to  now  the  ATaT  packet  network  service  has  had 
relatively  few  switches  with  high  throughput  and  networks 
with  few  hops.  The  1P88  will  have  software  for  tandea 
operation  by  the  third  quarter  of  this  year.  The  1P88  has  a 
four  level  congestion  control  strategy  which  is  essentially 
a  source  quench;  it  operates  on  virtual  circuits,  not  data- 
graas. 

The  5B88,  although  priasrily  a  voice  circuit  switch,  can 
provide  packet  switching  service.  Thus,  it  is  an 
operational  hybrid  switch.  Within  the  5B88  two  Micro- 
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processors,  Intel  8066s  and  Motorola  68000s  are  used  for 
interaediate  control  and  line  handling  respectively. 

7.1.3  H/A-COM  CP9000  Series  II 

The  N/A-COM  CP9000  Series  II  I MAC  8S1  is  also  a  miltiproces> 
sor  systea.  It  is  based  on  Intel  80286s;  Intel  80186 's  are 
used  as  port  processors.  Clusters  of  up  to  8  processors  are 
interconnected  on  a  high  speed  LAM  within  a  node.  Hardware 
redundancy  is  provided  for  reliability  and  the  X.25 
protocols  are  adhered  to.  Data  encryption  with  integral  DMS 
devices  is  available;  the  software  support  includes  a  fora 
of  TOS.  M/A-COM  believe  that  their  80286  and  specialized 
operating  systea  can  receive  B-1  or  B-2  Coapater  Security 
Center  certification.  Network  control  is  provided  on  a  VAX 
11/7S0. 

7.1.4  Northern  Telecoaa 

Northern  Telecoaa  has  just  introduced  their  DPM  series 
packet  switch;  this  is  an  upward  coapatible  version  of  their 
SL-IO  packet  switch  (MOB  8SI.  The  switch  architecture  is 
based  on  autipcoeessors  connected  via  high  speed  buses.  A 
full  redundancy  configuration  is  available  in  which  every 
user  has  access  lines  connected  to  two  independent  segaents 
of  the  switch.  These  segaents  are  in  turn  connected  to  two 
nodes  of  a  trunk  network.  The  high  end  DPM-SO  can  handle  up 
to  2880  access  lines  at  9.6  kbps,  or  96  access  lines  at  64 
kbps,  or  a  aixture  of  those  categories.  Up  to  30  trunks  at 
192  kbps  can  be  acconaodated.  The  DPM-SO  can  handle  up  to 
3000  packets  per  second  (of  unstated  size).  Other  neabers 
of  the  DPM  faaily  consist  of  sanller  packet  switches  ,  PADS, 
and  a  switch  especially  configured  for  trunk  and  gateway 
switching.  The  DPM  series  has  provision  for  high  speed 
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Interconnection  with  a  colocated  digital  voice  switching 
systeM,  thus  permitting  a  hybrid  switch  configuration. 

7.1.5  BBN 

In  addition  to  the  C/3x  series  of  packet  switches  BBN  has 
recently  Introduced  the  Butterfly  multiprocessor  system 
CBBM  85a,  BBN  85b].  This  multiprocessor  system  contains  a 
number  (4,  16,  64  or  256)  of  processor  nodes  which  are 
connected  to  the  same  number  of  comswn  meaM>ry  modules 
through  an  interconnection  network  t#hich  allows  any  pro¬ 
cessor  node  to  be  connected  to  any  memory  module.  Processor 
node-memory  SMdule  transfers  are  themselves  serial  packet 
transadasions  at  32  Mbps.  Sach  processor  node  contains  a 
Motorola  68000  series  processor,  local  memory  of  up  to  4 
Mbytes,  and  a  custom,  microprogrammed,  coprocessor  which 
acts  as  the  internal  packet  transmitter  and  receiver. 

The  Butterfly  processor  arose  out  of  a  design  for  a  high 
capacity  voice  talkspurt  multiplexer.  It  Is  also  being 
promoted  as  one  of  the  parallel  processing  alternatives  to 
supercomputers.  Small  Butterfly  configurations  are  being 
developed  for  use  as  data  communication  gateways  (MBS  84). 
Butterfly-based  packet  switches  will  replace  the  PLURIBUS  on 
DARPA*s  experimental  satellite  network. 

7.2  Future  Cosmsrclal  Packet  Switching  Developments 

Northern  Telecomm  has  plans  for  a  new  DPN-100  packet  switch 
with  twice  the  throughput  of  the  DPN-50.  Their  research 
arm.  Bell  Northern  Research  (MR),  has  been  experimenting 
with  more  flexible  protocols  aimed  at  satellite  trunking 
including  larger  window  sixes  and  provision  for  multi-packet 
retransmission  (DRY  851.  They  have  also  been  experimenting 
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with  Mixtures  of  satellite  and  terrestrial  trunks  between 
the  ssMB  node-pairs  and  provision  of  T08  routing  to  optlalze 
service  over  such  trunks  (CHU  851. 

Northern  TelecoMS  believes  that  the  future  of  digital 
coMsmnications  will  be  dominated,  especially  for  long  hauls, 
by  cheap  bandwidth  as  a  result  of  fiber  optics  developnents. 
As  a  result,  they  see  hybrid  packet-circuit  switches  sharing 
internode  trunks  as  principal  net%K>rk  conponents. 

ATAT  see  the  requirements  for  packet  switch  throughput 
growing  by  an  order  of  magnitude  every  five  or  six  years, 
thus  they  foresee  requirements  in  the  data  world  for  packet 
switches  to  handle  100,000  packets  per  second  before  the  end 
of  the  century.  Rven  so,  they  see  voice  as  the  major  driver 
in  their  telecommunications  world.  As  a  result,  they  seek 
maximum  coamMnality  bet%ieen  systems;  for  example,  the  1P88 
and  5P88  have  a  lot  in  cona»n.  There  is  some  indication 
that  ATAT  thinks  that  in  the  future  all  eonmunications  will 
be  digital  and  packet  In  nature.  Two  recent  papers  by 
Turner  (TUR  85,  TUR  86]  exemplify  this  concept.  The  first 
of  these  papers  discusses  the  use  of  switches  with  1.5  Obps 
throughput  to  handle  both  voice  and  data/  The  second  paper 
adds  in  broadcast  and  video  for  both  television  and 
conferencing.  In  this  paper  a  packet  switch  terminates  63 
fiber  optic  links  operating  at  100  Hbps. 
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Commercial  packet  switches  are  approaching  the  capacities 
and  trunk  and  line  speeds  required  for  the  MOPS.  They  also 
have  reliability  and  maintainability  features.  They  tend 
not  to  have  some  of  the  DoO/DCA  specific  requirements  such 
as  precedence.  8ome  switches  have  integrated  encryption. 
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At  least  one  switch  Is  belnq  proposed  for  coaputer  security 
certification. 

The  technical  questions  concerning  the  use  of  a  BK>difled 
coBSMtclal  switch  for  the  MOPS  is  the  extent  and  cost  of 
Modifications.  This  can  only  be  addressed  In  a  satisfactory 
way  by  the  vendors. 


46 


SPARTA,  INC. 


22  April  1966 


8.0  8TRATBOII8  POR  ACQUIRING  THB  NOPS 

There  are  only  two  posslbllltiee  for  acquiring  the  NGPS  froa 
the  vle«fpolnt  of  the  requlreaents.  Only  the  second  one  has 
an  iapact  on  the  requireaents  for  the  NGPS. 

a  new  DON  specific  packet  switch  (faally)  can  be 
designed,  developed,  and  produced. 

coaswrclal  packet  switches  can  be  procured  with 

aodificatlons  to  aeet  NGPS  requlreaents, 

i 

Alternative  ways  of  procuring  the  next  generation  DON  (in¬ 
cluding  the  NGPS)  such  as  leasing  versus  buying  do  not 
affect  the  requireasnts  for  the  NGPS.  The  technological 
pros  and  cons  of  the  two  acquisition  asthods  are  discussed 
briefly  below. 

8.1  Design,  Develop,  and  Procure  a  DDM  Specific  HOPS 

Proa  a  technological  viewpoint  this  aethod  will  guarantee 
that  all  of  the  specified  requlreaents  are  aet.  It  will 
result  in  receiving  exactly  what  is  wanted.  The  principal 
question  will  be  one  of  cost  effectiveness  for  a  procureaent 
of  the  order  of  500  to  1000  NGPSs. 

8.2  Buy  Modified  Conswrcial  Packet  switches 

As  noted  in  Section  7.3,  off-the-shelf  conswrcial  packet 
switches  will  not  aeet  all  of  the  requlreaents  for  the  NGPS. 
At  the  physical  level  they  are  not  likely  to  aeet  the 
TBMPB8T  requlreaents,  nor  will  they  have  the  desired  extent 
of  C0M8BC  equipaent  integration.  On^ the  protocol  level  they 
will  not  have  the  set  of  DDN  protocols  needed  for 
transition.  If  DDM  is  to  aake  the  transition  to  conswr- 
cially  based,  ISO  standard  protocols,  those  protocols  will 
alaost  certainly  have  to  be  aodified  to  allow  for  such  DDN 
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requirements  as  precedence  and  preemption.  i£  the  protocol 
Implementations  have  not  been  made  with  computer  security  In 
mind.  It  will  be  difficult,  if  not  impossible,  to  validate 
them  at  the  appropriate  level. 

If  there  is  interest  in  considering  the  acquisition  of 
comswrcially  baaed  switches,  it  would  be  valuable  to  publish 
the  HOPS  requirements  and  ask  for  comawnts  on  those  require- 
awnts.  At  that  tine,  some  of  the  difficulties  in  using 
conamrcial  switches  could  be  identified  and  a  more  detailed 
examination  could  be  made  of  tiays  to  cope  with  them. 
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9.0  SCHIDULB 

9.1  Date  MGPS  la  Needed 

Baaed  on  the  dlacuaalon  in  Section  3  it  appeara  that  the 
initial  inatallatlona  of  the  HOPS  will  be  needed  by  either 
the  beginning  of  1968  (no  C/300a  need)  or  the  beginning  of 
1991  C/300S  uaed).  That  ia*  by  the  latter  date*  aoam 
avitchea  will  require  greater  throughput  than  can  be 
provided  by  the  c/300.  Xnatallation  of  operational  M(3PSa 
iaq>liea  both  a  developaant  achedule  and  a  group  of 
additional  requlreaanta.  Soae  of  the  atepa  which  will  need 
to  be  taken  in  Making  the  MOPB  available  by  the  deaired 
date (a)  are  dlacuaaed  below.  Three  achedulea  for  the  HOPS 
are  praented  in  Schedulea  9-1  through  9-3  at  the  end  of  thia 
Section. 


9.2  Oevelopmnt  or  Adaptation  of  a  CoMierclal  Switch 

The  schedule  presented  for  the  1991  date  is  based  on  the 
developaent  of  a  DDN  specific  device*  the  alternative 
described  in  Section  8.1.  The  achedule  presented  for  the 
1988  date  assuaas  that  adaptation  of  a  coMwrcial  switch* 
discusses  in  section  8.2  is  the  procureaent  aathod;  this  is 
the  only  one  that  would  fit  that  tiaa  fraae.  If  c/300s  are 
brought  into  the  DON  and  the  1991  date  applies  it  would  be 
possible  to  use  a  less  intensive  schedule  for  adapting  a 
coaaercial  switch. 

9.3  Testing 


Concurrently  with  the  initiation  of  HOPS  developaent*  plan¬ 
ning  for  an  appropriate  test  bed  auat  soon  begin.  It  Is 
likely  that  the  present  test  bed  at  DCBC  can  be  the  basis 
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£or  the  MOPS  test  bed.  A  schedule  for  the  test  bed  progtSM 
is  included  in  each  of  Schedules  9>1  through  9-3. 

9.4  Other  RequiresMnts 

The  "Next  Generation  Network  Control  Center”  has  not  been 
examined  in  the  present  study.  Both  the  Future  Network 
Technology  Report  CHBR  8511  and  the  Draft  White  Paper  on  DDN 
Capacity  state  that  such  a  new  center  will  be  needed.  The 
requiresMnts  for  this  center  nust  be  generated  and  the 
necessary  centers  acquired  by  the  time  that  MOPS 
installation  takes  place. 

To  assist  in  making  the  decision,  "develop  or  adapt*  an 
early  call  for  coanwnts  should  be  made.  The  current  study 
provides  a  base  to  be  used  in  the  call  for  comments. 

9.5  Transition 

Transition  of  the  MOPS  into  DDN  will  require  an  installation 
schedule  which  is  coordinated  with  production  and  with  the 
installation  of  new  NOCs.  Candidate  sites  for  MOPS  would  be 
either  new  sites,  those  which  are  heavily  loaded  with 
traffic,  or  those  which  are  candidates  for  high  bandwidth 
trunks . 
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Schedule  9-1  Developaent  of  a  New  MOPS 

MOPS:  SS  87  ( 

Mew  DON  Plan  i  - 1 

Call  for  CoaeMicial  Consents i  * 

Final  Requlreaents  I  * — 

A  Specification  I  I  - 

B  Specification  I  I  - 

Publish  RPP  i  I 

Evaluate  Proposals  I  I 

Develop  Prototypes  I  I 

Test  Prototypes  I  I 

Begin  Production  I  I 

Develop  Transition  Plan  I  I 

Introduce  MOPS  Into  DDN  i  i 

HOPS  Test  Bed:  86  87 

Develop  Requireaents  I  ♦ — 

A  Specification  I  I  — 

B  Specification  I  I  — 

Publish  RPP  I  I 

Bvaluate  Proposals  I  I 

Build  Test  Bed  I  I 

Other  RequiresMnts:  86  87  I 

Plan  and  acquire  new  NOC  I  — ♦ - h 
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90  91 


90  91 
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schedule  9-2  Adaptation  o£  a  Comm 
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Schedule  9-3  Adaptation  of  a  Coaaercial  Switch  for  1991 


MOPS :  86 

Mew  DDM  Plan  I 

Call  for  CoMMrcial  CoMMntal 
Pinal  Requlreaenta  I 

A  Specification  I 

B  Specification  I 

Publish  RPP  I 

Bvaluate  Proposals  I 

Test  Adapted  Switch  I 

Develop  Transition  Plan  I 

Introduce  MOPS  into  DON  I 


S7  8S  89  90  91 

- 1  I  I  I  I 

•  I  I  I  I 

♦ —  till 

1*111 

I  I  I  I  ♦- 


MOPS  Test  Bed: 
Develop  RequiresMnts 
A  Specification 
B  Specification 
Publish  RPP 
Bvaluate  Proposals 
Build  Test  Bed 


91 


Other  RequiresMnts:  86  87  88  89  90  91 

Plan  and  acquire  new  MOC  I  - ♦ - ♦ - ♦ - ♦ - ♦- 
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10.0  RICOMHBMDATIONS 

This  Section  suBBurizes  the  requirements  for  the  HOPS 
developed  in  Section  6.  The  requirements  assume  a 
deployment  date  of  January  1991.  If  an  earlier  date  is 
chosen  as  regards  schedule  9.2,  then  the  MOPS  might  not  have 
to  meet  all  the  performance  requirements  if  it  can  be 
upgraded  by  1991.  Other  supporting  information  is 
presented  in  Section  3.6  and  Appendix  A. 


Date: 

Quantity: 

Performance: 


Ports: 


Production  models  available  January  1991 


Prom  500  to  1000 


Throughput  of  3360  750-bit  packets  per  second 
of  which  2560  are  for  tandem  traffic,  640  for 
originating  and  terminating  traffic,  and  160 
for  intranode  traffic. 


20  for  trunks  to  be  configured  for  line 
speeds  as  required  up  to  384  kbps. 


99  for  hosts  to  be  configured  for  line  speeds 
up  to  64  kbps 


DoD  Peatures:  Precedence,  Preemption 


Security: 


Integral  link  encryptors,  cryptographic 
checksumming  of  control  messages,  switch 
certified  at  C-2  level,  preferably  at  B-2. 
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coapatlbillty:  Must  be  able  to  interoperate  with  DON  PSMa  at 
the  Initial  release. 

User  Interfaces:  DOM  X.25,  X.75 

Operational  Peatores:  Congestion  Control,  Type  of  Service 
Routing,  Unattended  Operation,  Reaote 
Maintenance  Diagnosis,  Pail-soft  Architecture 


Reliability:  99.995%  uptiaw,  1000  hours  HTBP 
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APPBMDIX  A 

DISCUSSION  OP  PACKET  SWITCH  ARCHITBCTURB  AND  PBRPORMANCB 
A.O  INTRODUCTION 

This  Appendix  la  to  dlscass  how  salient  features  of  packet 
switch  architecture  affect  the  perfornance  of  that  packet 
switch,  how  packet  switch  throughput  is  Modeled,  and 
presents  an  exaMple  of  a  Multiprocessor  architecture  for  the 
NWS. 


A.l  Processor  Architecture  and  Packet  Switch  Perforsance 

A  siMple  Model  of  the  required  packet  switch  processing  is 
developed  and  used  to  evaluate  the  switch  architectural 
features . 

A. 1.1  Architectural  Features 

Salient  architectural  features  for  a  packet  switch  processor 
include  the  following: 

o  processor  instruction  set 
o  processor  serial  instruction  speed 
o  processing  eleMsnt  coMpleMsnt  (single,  dual. 
Multiple) 

o  priMsry  Meaory  organization 
o  MOMory-processor  interconnection 
o  priMary  MSMory  bandwidth 

An  assuMption  is  also  Made  that  individual  consMinication 
link  ports  are  handled  directly  by  specialized  processing 
subsystsMS.  These  subsysteMS  contain  local  storage  which 
can  be  block  transferred  to/froM  global  MOMory,  and  speci¬ 
alized  processors  for  synchronization,  perforMance  of  error 
checks,  and  for  rsMoval  or  addition  of  header  inforMation. 
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Prom  an  economic  standpoint  It  Is  desirable  1£  the  archi¬ 
tecture  utilizes  standard  "building  blocks"  such  as  micro¬ 
processor,  USARTS,  CRC  generator /checkers,  and  memory  de¬ 
vices.  The  performance  of  any  architecture  Is  a  function  of 
the  application  which  Is  to  be  run  upon  It.  In  order  to 
focus  the  analysis  of  packet  switch  architecture  for  this 
Appendix  the  packet  switch  application  has  been  amdeled 
simply  In  a  form  %fhlch  appears  to  be  very  useful  In  assess¬ 
ing  the  Importance  of  specific  architectural  features.  The 
numbers  proposed  In  the  model  are  believed  to  represent 
realistic  nominal  values.  Additionally,  they  provide  a 
basis  for  assessing  architecture/performance  relationship  in 
a  way  which  Is  not  strongly  affected  by  changes  In  the 
nominal  values  used  In  the  model. 


A. 1.2  Packet  Switch  Performance 


The  packet  switch  receives  and  transmits  an  approximately 
equal  number  of  bits  over  its  store-and-forward  channels. 
These  channels  are  both  those  to  and  from  directly  connected 
hosts  and  those  to  and  from  other  switches.  (The  packet 
switch  does  not  accumulate  Information  that  must  be  stored 
in  a  secondary  memory) .  The  packet  switch  successfully  uses 
queues  to  smooth  out  variations  In  packet  arrival  times  so 
that  Its  operation  can  be  viewed  as  being  In  a  steady  state. 


Given  that  the  bandwidth  of  the  cosmiunlcatlon  links  Is 
sufficiently  high,  the  rate  limiting  aspect  of  packet  switch 
operation  is  the  rate  at  which  decisions  and  memory 
management  can  be  performed  by  the  packet  switch.  Packets 
are  assumed  to  be  of  uniform,  1000-blt  size;  these  may 
represent  message  segments,  control  traffic,  aggregate 
acknowledgements,  etc. 
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The  model  for  processing  the  1000-blt  packets  is  simply 
that: 

a  decision  must  be  made  as  to  how  to  route  and 
otherwise  process  the  packet  <e.g.,  forward  it,  hold  it 
for  reassembly,  etc.); 

some  memory  management  SMist  be  performed  to  clear 
and/or  rearrange  memory  space. 

The  decision  process  is  assumed  to  require  an  average  of  500 
elementary  instructions  and  1000  bits  of  memory  must  be  ac¬ 
cessed  for  the  packet  memory  management.  With  this  process¬ 
ing  model  the  time  required  for  the  packet  switch  to  handle 
a  single  packet  is  the  sum  of: 

the  time  to  handle  2  block  data  transfers  between  local 
and  global  memory; 

the  time  to  execute  500  instructions; 

the  time  to  perform  memory  management  (taken  to  be  1 
instruction  per  %n)rd). 

Menmry  bandwidth  is  prominent  in  these  expressions  for  both 
data  transfers  and  for  the  rate  at  which  the  decision  in¬ 
structions  are  fetched  and  performed. 

The  second  term  above  is  related  to  both  the  processor 
serial  instruction  rate  and  the  instruction  set.  The  serial 
instruction  rate  can  be  expressed  in  instructions  per  unit 
time  (e.g.,  millions  of  operations  per  second  or  Mips).  The 
cycle  rate  of  the  processor  affects  its  speed  but  so  does 
the  processor's  internal  architecture.  For  example,  a  pro¬ 
cessor  with  a  large  register  set  can  concentrate  on  fast 
register-to-register  operations.  Thus  more  of  the  instruc¬ 
tion  mix  can  be  in  these  fast  operations.  Some  processors 
have  complex  operations  particularly  suited  for  packet 
switching  functions  such  as  enqueue  and  dequeue  instruc¬ 
tions.  These  instruction  types  can  significantly  reduce  the 
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average  nuaber  of  Instructions  required  for  the  packet 
switching  operations. 
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The  SMixlauM  throughput  for  this  Model  packet  switch  can  he 
laproved  by  supplying  additional  processors  to  perform 
decision  calculations  In  parallel  for  a  number  of  packets  at 
one  time.  The  rationale  Is  as  follo«f8:  Memory/  data  bus 
width  will  have  an  impact  on  the  first  and  third  terms 
above.  There  will  be  less  memory  access  required  to 
transfer  the  1000  bit  packets  If  wide  memories  and  data 
busses  (e.g.,  32-blt)  are  available.  Assuming  a  32-blt  word 
the  first  and  third  terms  will  require  about  100  meswry 
access  Instructions  and  the  processor  time  for  step  2 
Instructions  will  predominate. 

In  the  situation  just  examined  the  ratio  of  decision  In¬ 
structions  to  data  movement  Instructions  was  5:1.  Using  5 
processors  would  bring  the  decision  processing  time  down  to 
the  data  movement  time  representing  a  more  balanced  situa¬ 
tion.  However,  each  processor  would  require  Its  o«m 
instruction  stream  and  thus  global  memory  access  and  data 
paths  could  become  the  new  bottleneck. 
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In  sonury,  a  specific  packet  switch  architecture  can  be 
exaained  with  the  following  steps: 

identify  a  packet  size; 

postulate  the  nunber  of  processing  steps  for  one 
packet; 

postulate  requlresMnts  for  data  aoveaent  and  mmoxy 
■anagesMnt; 

use  the  processor  specifications  for  sMswry  bandwidth 
and  Instruction  tlsM  to  find  the  tlM  required  to 
process  one  packet;  Invert  to  find  mmximam  packet  rate; 

exaalne  effects  of  Increased  sMeory  bandwidth  and 
faster  Instruction  execution; 

exaelne  ways  to  balance  decision  instruction  ties  with 
data  BK>veaent  tlee 

Two  Important  factors  which  are  not  related  to  packet  switch 
throughput  were  not  taken  Into  account  in  this  discussion. 
The  first  factor  Is  the  ability  of  a  aultl processor  system 
to  provide  graceful  degradation  If  one  or  more  processors 
fall.  This  ability  can  only  be  achieved  wit  an  appropriate 
operating  system  (note  that  we  are  not  referring  to  a  gener¬ 
al  purpose  operating  system  but  a  limited  purpose  operating 
system  for  the  specific  packet  switch  application).  The 
second  factor  Is  the  problem  of  providing  a  certain  level  of 
computer  security  In  the  packet  switch.  Both  of  these  fact¬ 
ors  are  addressed  In  the  discussion  of  overall  packet  switch 
requirements.  They  are  likely  to  be  Important  In  the  selec¬ 
tion  of  specific  processors  for  a  multiprocessor  system. 
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A. 1.3  Multiprocasslnq/Parallel  Architectures 

The  discussion  in  the  previous  section  has  wide  a  strong 
i«plication  that  Multiprocessor  architectures  are  a  desi¬ 
rable  direction  for  the  next  generation  packet  switch.  At 
the  saM  tiae,  it  is  clear  that  the  interconnection  of  proc¬ 
essors  and  MBM>ries  is  a  key  issue  if  bottlenecks  are  to  be 
avoided.  In  the  exaaple  of  the  previous  section  the  use  of 
5  processors  reaoved  the  bottleneck  having  to  do  with  the 
ratio  of  decision  instructions  to  data  aovoMent  instruc¬ 
tions.  This  was  accomplished  at  the  expense  of  a  potential 
bottleneck  in  the  global  memory  and  data  paths. 

There  are  at  least  two  ways  in  %fhich  this  potential  bottle¬ 
neck  can  be  avoided.  A  current  topic  of  major  interest  in 
parallel  computation  is  the  provision  of  mnltiple,  high 
bandwidth,  paths  between  processors  and  memories.  (A  recent 
collection  of  papers  on  this  topic  will  be  found  in  an  XIII 
Tutorial  volume  (VUC  651).  A  less  conventional  solution  can 
be  envisioned  in  a  system  idiose  data  storage  consists  of 
shift  registers  and  PZPO  buffers,  resulting  in  a  special 
purpose  processing  system. 

A.  1.4  Susmuiry  of  the  Model 

This  sisplified  discussion  of  a  model  for  packet  switch 
performance  has  highlighted  the  following  steps: 

1.  Identify  a  basic  model  of  the  processing  performed 
by  a  packet  switch;  here  a  1000  bit  data  segawnt  %ms 
used. 

2.  Postulate  the  steps  required  to  process  basic 
event(s);  here  500  elementary  instructions  %mre  assumed 
necessary  for  decisions. 

3.  Postulate  requirements  for  data  movement  and  memory 
management . 
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4.  Uae  processor  specifications  for  Memory  bandwidth 
and  Instruction  timing  to  assess  time  required  to 
process  a  segment  of  data;  invert  to  obtain  the  maximum 
data  rate. 

5.  Examine  the  effect  of  increased  memory  bandwidth, 
either  through  faster  access  times  or  wider  data  paths. 

6.  Ixamina  the  effect  of  faster  instruction  execution. 

7.  Ixamine  the  ratio  of  decision  instruction  execution 
time  to  data  movement  time. 

8.  The  steps  above  explore  the  domain  in  which 
processing  is  the  limiting  factor;  also  explore  the 
internal  limiting  data  bandwidth  as  defined  by  memory 
and  data  path  bandwidth. 
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A. 2  Network  Throughput  and  Packet  switch  Throughput 


A. 2.1  Top  Level  Model 


The  underlying  packet  switch  aodel  views  the  switch  as  a 
node  In  a  network  of  servers.  The  switch's  outgoing  tele- 
cossnanicatlon  links  are  its  own  net%fork  servers.  Incoalng 
telecoswunl cat Ions  links  are  the  paths  for  Introducing  new 
service  requests.  This  aodel  is  Illustrated  below. 


The  load  experienced  by  a  packet  switch  In  a  network  will 
depend  upon  the  following  network  characteristics: 

1.  The  set  of  traffic  flow  demands  between  all 
possible  network  entry  and  exit  points  (l.e.,  nodes 
that  have  hosts  attached).  The  heavier  these  flows 
are,  the  heavier  the  load  experienced  by  the  packet 
switches. 

2.  When  traffic  flow  demands  are  very  unbalanced  and 
the  allocation  techniques  are  based  upon  minimizing 
delay,  some  network  paths  may  deviate  significantly 
from  shortest  hop  count  paths  to  avoid  overloading. 

3.  The  capacities  of  the  communication  links 
Interconnecting  the  node  limit  the  number  of  bits  that 
can  flow  through  a  packet  switch. 

4.  The  topology  of  the  Interconnections  of  packet 
switches.  A  packet  switch  with  very  few  links  is  apt 
to  experience  a  lighter  load  than  one  with  many 
incoming  and  outgoing  'Inks. 
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A. 2. 2  IstlMtlon  Techniques 

Given  the  wealth  of  paraaetera  that  determine  the  load  on  a 
packet  switch,  an  accurate  yet  simple  load-predicting  model 
Is  not  obtainable.  Instead,  one  of  several  estimation  tech¬ 
niques  may  be  used  to  predict  packet  switch  loading.  Compu¬ 
ter  system  engineers  agree  upon  a  hierarchy  of  estimation 
techniques  for  loads  and  performance  of  computer  systems 
subject  to  multiple  job  streams  of  variable  Intensity  (such 
as  the  nodes  of  a  computer  network). 

The  most  accurate  estimates  come  from  benchmarks  In  which 
the  actual  computer  systems  are  run  under  actual  loads  while 
measurements  are  being  made.  This  also  allows  distributions 
rather  than  single  estimates  to  be  made  for  node  loading  and 
performance. 

Accurate  estimates  of  system  performance  are  frequently  made 
from  discrete  event  sllwilatlons.  These  are  computer  pro¬ 
grams  often  constructed  using  specialized  languages  repre¬ 
senting  processes  and  series  of  random  events  treated  by 
those  processes.  Simulations  model  a  large  number  of  system 
features  In  such  m  way  as  to  represent  their  Interactions  In 
real  time.  (Real  tlow  Itself  Is  simulated  by  a  clock  that 
Is  advanced  only  under  contriqliil^f  the  simulation  program.) 
Discrete  event  slmulatlofr^^ograms  can  model  a  large  number 
of  events  to  report  '‘'Statistics  about  loading  and  perform¬ 
ance,  rather  than  single  estimate  answers. 

Satisfactory  estimates  can  be  made  from  queuing  theory 
calculations  based  upon  the  arrival  frequencies  of  jobs 
(l.e.,  frequency  of  arrival  of  packets  over  a  line)  and  upon 
the  probability  distribution  of  job  service  times  (l.e.,  the 
distribution  of  packet  lengths  divided  by  the  link's  bit 


SPARTA,  INC. 


22  April  1986 


rate).  In  coaputer  networks  there'  Is  (usually)  an  Inter¬ 
action  between  the  path  allocation  technique  and  the  queuing 
delays  estlnated  at  each  node.  The  allocation  technique 
dynamically  adapts  to  the  delays  experienced  at  each  node, 
rerouting  £lo%rs  a%iay  from  congested  nodes.  Algorithms  for 
evaluating  network  node  load  and  performance  can  take  Into 
account  the  allocation  technique  at  the  expense  of 
computational  complexity.  (Section  B.3  below  describes 
steps  in  estimating  detailed  network  loading  from 
topological  descriptions  and  flow  requirements.  Loading  and 
performance  of  individual  nodes  can  be  derived). 

Adequate  estimates  can  often  be  made  using  "rules  of  thumb". 
In  the  case  of  networks,  such  rules  of  thumb  include 
judgments  about  the  average  number  of  hops  in  a  network  path 
and  the  implicit  assumption  that  every  path  constrains  this 
number  of  hops,  and  about  the  ratio  of  store-and-for%Mird 
traffic  (from  other  packet  switches)  to  newly  entering 
traffic  (from  attached  hosts).  These  rules  of  thumb  are 
derived  from  experience  with  benchmarks  and  from  more 
detailed  analytical  or  discrete  event  simulations.  As  such 
they  represent  "average"  or  "typical"  values.  They  do  fail 
to  represent  the  variance  beyond  "typical"  cases  and  as  a 
result  the  possible  consequences  of  this  variance.  (In 
contrast,  analytic  queueing  theory  calculations  do  calculate 
the  consequences  of  variance  in  flow  demands,  packet 
lengths,  etc.) 

For  the  purpose  of  illustrating  sample  packet  switch  per¬ 
formance  requirements  «fe  have  used  rule-of-thumb  calcula¬ 
tions.  These  are  easier  to  follow,  yet  they  are  supported 
by  experience  with  the  mote  detailed  calculations. 

A.2.3  Iterative  Techniques  for  Network  Load  Leveling 
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This  section  describes  an  al^orlthsi  for  assigning  flows 
(l.e.  of  data)  over  a  net%rork  of  packet  switch  nodes  and 
coBSMinlcation  links.  The  procedure  is  briefly  presented 
here. 

Plxst,  an  initial  flow  assignsMnt  is  atteapted  by  seeking 
the  shortest  path  (in  terns  of  nunber  of  hops)  for  each  flow 
desMnd.  The  technique  tor  balancing  flows  in  the  initial 
asslgnsMnt  attespt  (and  in  subsequent  refinenents)  is  to  use 
per  link  delay  estiMtes,  based  upon  queueing  theory,  rather 
than  nunber  of  hops  in  path  length  calculations.  The  sane 
link  aay  be  used  for  aore  than  one  flow  and  say  be  over 
utilised.  In  such  a  case,  an  attenpt  suist  be  suide  to 
balance  the  flow  so  that  no  link  is  over  utilized. 

Next,  the  success  of  the  initial  assignment  is  evaluated 
according  to  link  utilization.  Whenever  a  flow  assignaent 
over  utilizes  a  link,  the  flo«ni  are  hypothetically  rescaled 
to  a  utilization  arbitrarily  jusb^nder  100%  (e.g.  99%). 
This  has  an  ^Uaet  upon  the  queueing  theory  calculations  In 
a  next  iteration  —  very  large  delays  are  calculated  for  the 
link  in  question.  Consequently,  ainiaua-delay  paths  will 
avoid  the  link  in  question,  thus  re-balancing  the  load. 

Finally,  repeated  attempts  are  made  to  optisally  balance  the 
load  so  as  to  minimize  a  grand  delay  average.  Solution 
convergence  is  used  to  halt  the  iterations.  However,  this 
does  not  guarantee  a  globally  optimum  flow  assignment. 

This  technique  involves  iterative,  compute-intensive  opti¬ 
mization  and  only  partially  represents  how  a  distributed 
routing  algorithm  performs  its  own  flow  assignments.  (A 
distributed  routing  algorithm  performs  at  each  node  a 
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shortest  path  calculation  based  upon  per-llnk  delay  esti¬ 
mates  that  are  received  at  each  node).  It  also  does  not 
provide  a  %Miy  o£  evaluating  dynamic  aspects,  such  as  the 
time  required  to  adapt  to  network  configuration  changes.  It 
does  provide  a  hypothetical  "performance  envelope”  of  net¬ 
work  delays,  against  %fhich  the  performance  of  a  routing 
algorithm  can  be  judged.  It  can  also  be  used  to  evaluate 
the  maximum  traffic  handling  capacities  of  hypothetical 
net«rork  configurations. 
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A. 3  An  sxaaple  Architecture  for  the  hops 

Zn  this  Section  we  esteblish  that  an  econoaical  and  elegant 
architecture  can  be  found  to  satisfy  the  requireaents  of  the 
Next  Generation  Packet  Switch.  Factors  that  need  to  be  taken 
into  account  for  the  architecture  include:  throughput, 
reliability,  security,  flexibility. 

These  factors  are  discussed  in  general  tersa  below. 

A.  3 . 1  Throughput 

A  anltiprocessor  architecture  is  appropriate  for  the  MOPS  as 
the  following  discussion  aakes  clear.  A  requlreaant  %fas 
developed  in  Section  3.6  that  approxiaately  3360  packets  per 
second  be  handled.  The  aajority  of  these  are  tandea  traffic 
and  we  can  use  the  figure  of  500  Instructions  per  packet, 
developed  in  A.l  as  the  ^oainant  factor.  Therefore,  we  need 
to  execute  over  1.6  ailHon  instructions  per  second  (MZPs). 
A  good  goal  would  be  2  MZPs.  The  nature  of  the  instructions 
and  of  the  data  handling  operations  aust  also  be  taken  into 
account.  The  way  in  which  current  coaputing  architectures 
provide  increased  po%fer  at  lower  cost  is  the  use  of  parallel 
processing  when  the  coaputing  tasks  can  be  partitioned.  The 

'  a 

coaputing  tasks  of  a  packet  swi^<0‘are  just  such  a  case;  an 
aggregate  of  processors  w^jpet  total  coaputing  power  will  be 
about  2  MZPs  can  handlpl^^The  throughput.  We  aight  do  well  to 
double  this  requireaent  as  a  safety  factor  and  to  allow  for 
additional  features  to  be  provided. 

A.  3. 2  Reliability 

Although  coaputing  coaponents  continue  to  becoae  individual¬ 
ly  aore  reliable,  the  MOPS  requires  a  very  high  degree  of 
reliability.  Purther,  failure  of  one  coaponent  should  not 
disable  the  switch.  That  is,  the  systea  should  be  fault- 
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tolerant  or  fall-soft.  As  in  the  exaB4>le  of  the  PLURIBUS 
and  the  Butterfly,  a  multiprocessor  architecture  with 
appropriate  interconnections  is  %rell  suited  to  such  a 
requirement. 

A.  3. 3  Flexibility 

The  next  generation  packet  switch  will  have  to  support  a 
dynamic  communication  world.  The  current  generation  of 
switches  has  evolved  steadily  in  terms  of  its  interfaces  and 
internal  operations,  but  al%#a]^  on  the  same  basic  archi¬ 
tecture.  The  use  of  a  stored  program  computer  has  permitted 
such  flexibility;  this  facility  can  be  realized  in  a  multi¬ 
processor  switch.  Multiprocessor  operating  systems  are  not 
as  mature  as  those  for  well  established  uniprocessors; 
hoiMver,  orderly  development  methods  for  such  operating 
systems  are  currently  %ni11  in  hand. 
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A.  3.4  Security 

The  multi processor  system  postulated  here  for  the  HOPS  Is  a 
distributed  computer  system  or  a  net%N>rk  of  computers. 
Assuring  computer  security  for  distributed  systems  or  for 
networks  of  computers  Is  still  in  a  primitive  stage.  The 
National  Computer  Security  Center  has  not  yet  released  their 
final  version  of  the  DoD  Trusted  Net«#ork  Security  Criteria 
(the  draft  is  (CSC  851).  Nevertheless,  an  orderly  and 
formal  approach  computer  security  can  be  taken  in 
developing  a  multiprocessor  based  N(98. 

A. 3. 5  Summing  Up 

This  Section  has  sho%m  that  a  miltlprocessor-based  solution 
to  the  hardware  architecture  of  the  NOPS  is  realistic  and 
attractive.  The  critical  point  for  throughput  of  that 
solution  Is  how  the  processors  wor1S  with  the  aggregate  of 
memory;  that  is,  the  nature  of  the  processor-memory  bus 
structurm^and  performance.  This  solution  is  also  attract¬ 
ive  to  commercial  vendors  as  the  examples  In  Section  7.1 
have  sho«m. 
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APPENDIX  B  >  ALTERNATIVE  NETWORK  ENVIRONMENTS 
B.O  Introduction 

In  Section  5  we  presented  an  environment  for  the  N6PS  «rtilch 
can  best  be  described  as  a  scaling  up  of  the  DDN  as  It 
exists  today.  Topologically,  the  net%rork  would  not  be 
changed  very  much  and  the  average  hop- length  would  be  about 
the  same.  Differences  would  be  mainly  in  volumes  of  traffic 
carried.  In  accommodation  to  new  communication  technologies, 
and  in  provision  of  more  specialization  such  as  Type  of 
Service. 

In  this  Appendix  we  discuss  briefly  some  alternatives  to 
that  topology  and  environment  for  the  NGPS.  These  possi¬ 
bilities  received  a  very  modest  amount  of  attention  during 
the  development  of  this  report.  The  Introduction  to  Section 
5  presents  the  rationale  for  electing  continuation  of  the 
"classical**  topology.  Nevertheless,  for  completeness,  we 
describe  here  some  alternatives  which  differ  from  that 
"classical"  topology.  These  alternatives  are  described  in 
Section  B.l  below.  The  probable  effects  of  the  alternatives 
on  NOPS  requirements  are  discussed  in  Section  B.2. 

The  Future  Network  Technology  Study  identified  two  ways  of 
using  high-speed  trunks  in  DDN  (pp  236-37  of  IHER  85a]). 
One  of  them  is  the  hierarchical  network  in  B.l. 2  below.  The 
other  multiplexes  T1  circuits  into  64  kbps  trunks  and  limits 
individual  trunks  to  that  speed.  We  have  rejected  the 
second  option  in  Section  5  of  this  report. 

We  believe  that  the  alternatives  discussed  below,  as  well  as 
some  which  have  not  yet  been  "invented",  represent  possi- 
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blllties  which  will  need  to  be  considered  In  the  generation 
which  follows  the  NOPS. 

B.l  Alternative  Strategies  for  the  DON 

In  this  section  a  nustber  of  alternative  archltectures/envl- 
ronaents/strategies  for  the  DOM  of  1990-f  are  briefly  presen¬ 
ted.  They  all  allow  for  the  substantial  gro%rth  In  user  sup¬ 
port  requirements  developed  In  Section  3,  though  they 
represent  substantially  different  ways  of  getting  there. 
Bach  description  of  an  alternative  also  contains  a  very 
concise  discussion  of  the  "pros*  and  "cons"  for  such  a 
systea. 


B.1.1  DDM  as  an  Internet 

Section  5  developed  a  concept  of  DDM  as  a  continuation  of 
the  current  DOM  In  which  the  net  is,  at  least  topologically, 
a  uniform  whole.  That  Is  the  net  can  be  conceptually 
diagramed  as  shown  In  Figure  B-1  (a  duplicate  of  Figure  5-1, 
but  shown  here  for  convenience).  An  alternative 
architecture  for  the  DDM  would  be  the  provision  of  many 
Independent  net%rorks  provided  or  maintained  by  groupings  of 
users  %fhlch  fall  together  logically.  The  majority  of  the 
traffic  In  these  Independent  networks  would  be  Intra¬ 
network.  The  networks  would  be  Interconnected  by  a  DDM 
which  provided  only  Internet  service.  Figure  B-2  Is  a 
diagram  of  that  configuration. 
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B.1.2  DON  as  an  Hierarchical  Network 

This  version  of  a  DON  can  be  considered  as  a  physical  Imple- 
■entatlon  of  the  concept  of  area  routing  for  DON.  Such  a 
network  la  Illustrated  conceptually  In  Figure  B-3.  As  far 
as  groupings  of  users  are  concerned.  It  is  soM%fhat  like  the 
diagram  of  Figure  B-2.  There  are  two  major  differences  In 
the  hierarchical  concept:  the  groupings  are  not  connected  to 
each  other  through  a  gatetmy  but  through  a  some%rhat 
specially  configured  packet  switch;  the  nodes  are  all 
supported  by  DON. 
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B.1.3  DDM  as  a  Multi-Vendor  Network 

This  alternative  assusMS  that  It  will  be  possible  to  rely  on 
that,  at  most,  minor  adaptations  of  commercial  packet 
switches  will  be  necessary  to  provide  the  functionality 
needed  by  DDM.  It  further  supposes  that  different  vendors 
will  be  chosen  for  successive  acquisitions  and  further 
assumes  that  standardization  will  continue  to  take  place  to 
the  extent  that  switches  from  different  vendors  will  be  able 
to  be  Interconnected  without  difficulty.  (If  not,  net%rorks 
of  different  vendors  can  al%(ays  be  connected  through  an 
Internet.)  Under  these  conditions  it  will  be  possible  to 
add  new  switches  %fhlch  represent  the  best  value  at  the  time 
of  acquisition  and  older  equipment  will  be  able  to  coexist 
with  newer.  Plgure  B-4  represents  such  a  network. 
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As  a  practical  matter,  the  adaptations  to  commercial  switch¬ 
es  %rhlch  will  need  to  be  made  will  represent  a  substantial 
software  effort.  Since  the  standards  met  by  such  switches 
will  be  at  an  Interface  level,  use  of  multiple  vendors  will 
Increase  the  soft%rare  adaptations  by  the  number  of  vendors. 

B.1.4  DDM  as  a  Maximal  Backbone 

This  alternative,  diagramed  In  Figure  B-5,  would  provide  a 
network  with  relatively  few  nodes.  Bach  node  would  serve 
many  more  subscribers  than  present  nodes.  This  maximal 
backbone  Is  equivalent  to  the  top  tier  In  the  hierarchy 
discussed  In  B.1.2.  It  Is  also  the  same  as  the  top  tier  In 
the  high  speed  backbone  of  the  Future  Network  Technology 
Study  (Figure  4.12,  p  237  of  (HBR  85b].  Such  nodes  would 
represent  a  more  significant  target  for  physical  denial  of 
service.  Therefore,  for  reliability  and  survivability 
reasons  It  would  be  necessary  for  subscribers  to  be  homed  to 
two  different  nodes.  The  nodes  themselves  would  be  Inter¬ 
connected  by  high  data  rate  communication  links.  From  the 
standpoint  of  the  network,  this  configuration  would  be  simi¬ 
lar  to  an  upscaling  of  the  present  DON;  the  nodes  would  have 
greatly  Increased  throughput  and  many  more  user  ports. 
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B.l.S  DON  as  a  Hybrid  Network 

This  alternative  Is  rather  different  fron  those  described 
above.  The  others  (Sections  Bl.l  through  B1.4)  are  all  data 
networks  and  this  architecture  represents  a  network  for 
data,  voice,  and  perhaps  other  transmissions  as  %iell.  Such 
a  network,  shown  In  Figure  B>6,  would  provide  a  flexible  use 
of  Inter-node  trunks.  These  trunks  would  be  allocated  to 
data  or  to  voice  In  a  time-varying  mixture. 
Technologically,  this  would  be  In  accordance  with  the 
Intended  development  of  ISDN.  The  packet  switch  viould  have 
to  take  on  new  functionality  as  it  reallocated  trunk 
capacity.  In  principle,  this  %fould  be  a  powerful  and 
flexible  way  to  maximize  communication  trunk  resources.  In 
practice  we  probably  do  not  know  enough  at  this  time  to  make 
Intelligent  plans  for  such  usage. 
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There  are  a  nunber  of  problems  with  such  a  system  for 
DoD/DCA  usage  at  this  time.  Although  there  are  commercial 
hybrid  voice/data  packet  switches  (for  example  the  No.  5  ESS 
described  in  section  7.1),  these  switches  utilize  separate 
voice  signaling  and  they  do  not  necessarily  have  any 
provision  for  DoD  quality  communication  security  of  either 
end-to-end  or  traffic  flow  security. 

Burst  switching,  another  alternative  to  packet  or  circuit 
switching  has  been  proposed  recently  for  voice  transmission 
(HAU  831.  Burst  switching  is  a  form  of  very  fast  circuit 
switching.  Bach  time  a  talkspurt  starts  a  new  circuit 
allocation  is  made.  This  lasts  for  the  length  of  the 
talkspurt  tdiich  is  typically  a  few  seconds. 

B.2  The  Effects  of  the  Alternatives  on  NGPS  Requirements 


The  conversion  of  DON  to  an  internet  would  require  less 
switches.  These  might  not  need  to  be  of  higher  performance 
than  the  C/300s.  They  would  require  new  software,  a  major 
cost  in  any  PSN  procurement.  The  hierarchical  and  maximal 
backbone  environments  would  require  a  moderate  number  of 
very  high  performance  switches.  A  multi-vendor  network 
would  mean  accommodating  DON  to  modifications  of  commercial 
packet-switching  practice.  The  hybrid  network  could 
possibly  capitalize  on  commercial  offerings  %rhich  may  arise 
out  of  ISDN. 
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